Mercurial > dovecot > original-hg > dovecot-1.2
changeset 7798:8e206e25a142 HEAD
Merged latest v1.1 changes.
line wrap: on
line diff
--- a/.hgtags Sat May 17 17:50:54 2008 +0300 +++ b/.hgtags Mon Jun 09 05:11:18 2008 +0300 @@ -20,3 +20,8 @@ 958e377fc1dcdc88be2f8cfaba68fc5ad4c15dc6 1.1.rc1 dfd811aa0418000e2b60d9cafb67817cdc2dbdb9 1.1.rc2 c73d6224a96b3c07740dd9880cfe1b066e6238ff 1.1.rc3 +4607141a6bdc4226acbbece913d9780e5c3c818f 1.1.rc4 +3b09af6458e590f5b05fba89847a80441b6c0d5f 1.1.rc5 +79857a116d281846441c866ae2a9f8f2e4d7b0cf 1.1.rc6 +290bd8b0c2d76f6fccd3add14bb3007c86ce34ff 1.1.rc7 +58c7f5c31db1801d9c7495354e6911da898aaa65 1.1.rc8
--- a/NEWS Sat May 17 17:50:54 2008 +0300 +++ b/NEWS Mon Jun 09 05:11:18 2008 +0300 @@ -1,3 +1,84 @@ +v1.1.rc8 2008-06-03 Timo Sirainen <tss@iki.fi> + + + deliver: Added -p parameter to provide path to delivered mail. + This allows maildir to save identical mails to multiple recipients + using hard links. + - rc6/rc7 broke POP3 with non-Maildir formats + - mbox: Saving a message without a body or the end-of-headers line + could have caused an assert-crash later. + - Several dbox fixes + +v1.1.rc7 2008-05-30 Timo Sirainen <tss@iki.fi> + + - Fixed compiling problems with non-Linux OSes + +v1.1.rc6 2008-05-30 Timo Sirainen <tss@iki.fi> + + * Index file format changed a bit. If an older Dovecot v1.1 reads + index files updated by rc6+, they may give "Invalid header record + size" or "ext reset: invalid record size" warnings. v1.0 won't give + these errors. + * IMAP: LIST .. RETURN (X-STATUS) command return now LIST entries + before STATUS entries. + * zlib plugin: Uncompress if the message begins with zlib header + instead of looking at the 'Z' flag. This fixes copying with hard + links. Based on a patch by Richard Platel. + + + IMAP: SORT index handling code was half-rewritten to fix several bugs + when multiple sessions were sorting at the same time. The new code is + hopefully also faster. + + Maildir: If POP3 UIDL extra field is found from dovecot-uidlist, + it's used instead of the default UIDL format (or X-UIDL: header). + This allows easily preserving UIDLs when migrating from other POP3 + servers. Patch by Nicholas Von Hollen @ Mailtrust. + + Maildir: ,W=<vsize> is now always added to maildir filenames + + deliver: Avoid reading dovecot-uidlist's contents if possible. + + Added %T modifier = Trim whitespace from end of string + - IMAP: Fixed some bugs in LIST-EXTENDED implementation. + - IMAP: If client tries to change the selected mailbox state while + another command is still running, wait until the command is finished. + This fixes some crashes and other unwanted behavior. + - allow_nets userdb setting was broken with big endian CPUs + +v1.1.rc5 2008-05-05 Timo Sirainen <tss@iki.fi> + + + Support cross-realm Kerberos 5 authentication. Based on patch by + Zachary Kotlarek. + + Added dict_db_config setting to point to a Berkeley DB config file. + + If mail_chroot ends with "/.", remove chroot prefix from home + directory. + - Fixed several bugs and memory leaks in ACL plugin. LIST and LSUB + may have listed mailboxes where user had no 'l' access. STORE could + have been used to update any flags without appropriate access. + - mbox: Valid-looking From_-lines in message bodies caused the message + to be split to two messages (broken since v1.0). + - Plugin initialization hooks were called in wrong order, possibly + causing problems when multiple plugins were used at the same time. + - Expire plugin was broken + - LIST-EXTENDED options were ignored. + - LDAP: Static attribute names weren't working correctly + - deliver: mail_uid and mail_gid settings weren't used. + - pop3 + maildir++ quota: maildirsize file wasn't created if it + didn't exist already. + - dnotify: Waiting for dotlock to be deleted used 100% CPU + +v1.1.rc4 2008-04-01 Timo Sirainen <tss@iki.fi> + + * Fixed two buffer overflows in str_find_init(). It was used by + SEARCH code when searching for headers or message body. Added code + to catch these kind of overflows when compiling with --enable-debug. + Found by Diego Liziero. + + + LDAP: Added debug_level and ldaprc_path settings (OpenLDAP-only) + + Squat: Added fts_squat = partial=n full=m settings. See the wiki. + - dbox metadata updating fixes. + - quota: backend=n didn't work + - SEARCH RECENT may have returned non-recent messages if index files + were created by v1.0. + - If mailbox was opened as read-only with EXAMINE, STOREs were + permanently saved. + - LDAP: Templates were somewhat broken (by richs at whidbey.net) + v1.1.rc3 2008-03-09 Timo Sirainen <tss@iki.fi> * Fixed a security hole in blocking passdbs (MySQL always. PAM, passwd
--- a/TODO Sat May 17 17:50:54 2008 +0300 +++ b/TODO Mon Jun 09 05:11:18 2008 +0300 @@ -1,20 +1,27 @@ - - dbox: - - "metadata changed unexpectedly" with alt paths - why? - - check that metadata is always correct and whitespace contains only - whitespace - - "File unexpectedly lost" doesn't get fixed by itself - - doesn't call fsync - - do something about From_-lines splitting mails with mboxes + - recent assert. both with mbox and maildir. + - sieve-cmu.c crash: i_assert(buf->used - 1 == part->body_size.physical_size); + - convert plugin: Create a r/w lock for a file. It's read-locked if + conversion isn't wanted and released when process dies. If conversion is + wanted and write-lock succeeds, conversion is done, if write-lock doesn't + succeed it fallbacks to using the old storage. When process is exiting it + again tries to write-lock and do the conversion. Add a parameter that + specifies if conversion should be done. - lucene: handle replacement chars? - squat: - wrong indexid - fts_build_init() assertion failed: (last_uid < last_uid_locked) - - nfs support (cache flushes, how can write fail with ESTALE?) - is locking done right? it reads header without file being locked? - split after ~8 bytes? - expunges are delayed until more mails are added - test replacement chars (SEARCH / SORT / Squat) + - dbox: + - "File unexpectedly lost" doesn't get fixed by itself + - Fix support for multi-message files + - Delete dovecot-keywords and dovecot-uidlist after all maildir files + have been converted to native dbox + - DEBUG: buffer overflow checking code probably doesn't handle a successful + t_try_realloc() or pool_alloconly_realloc() properly - cache: compress when we can drop temporary fields. - POP3 UIDL caching - ACL: "foo/bar" in public namespace -> LIST "" % doesn't show "foo" @@ -22,6 +29,12 @@ - auth_gssapi_hostname = %Xl - proxying would also want DNS lookups, but not reverse.. - ldap domain lookups which set the base for user lookup + - ldap: same attribute can't be used for multiple values. + - ldap: multiple attributes can't be merged to a single value. + + - Per-user options: + - Deny deleting non-empty mailboxes + - Disable IDLE "still here" notifications - maildir+pop3/deliver fast updates: - with locking enabled, pop3 could just keep the one and same sync lock and
--- a/configure.in Sat May 17 17:50:54 2008 +0300 +++ b/configure.in Mon Jun 09 05:11:18 2008 +0300 @@ -437,7 +437,7 @@ have_ioloop=no if test "$ioloop" = "best" || test "$ioloop" = "epoll"; then - AC_CACHE_CHECK([whether we can use epoll],epoll_works,[ + AC_CACHE_CHECK([whether we can use epoll],i_cv_epoll_works,[ AC_TRY_RUN([ #include <sys/epoll.h> @@ -446,12 +446,12 @@ return epoll_create(5) < 1; } ], [ - epoll_works=yes + i_cv_epoll_works=yes ], [ - epoll_works=no + i_cv_epoll_works=no ]) ]) - if test $epoll_works = yes; then + if test $i_cv_epoll_works = yes; then AC_DEFINE(IOLOOP_EPOLL,, Implement I/O loop with Linux 2.6 epoll()) have_ioloop=yes ioloop=epoll @@ -489,7 +489,7 @@ if test "$notify" = "" || test "$notify" = "inotify" ; then dnl * inotify? - AC_CACHE_CHECK([whether we can use inotify],inotify_works,[ + AC_CACHE_CHECK([whether we can use inotify],i_cv_inotify_works,[ AC_TRY_RUN([ #define _GNU_SOURCE #include <sys/ioctl.h> @@ -523,12 +523,12 @@ return 0; } ], [ - inotify_works=yes + i_cv_inotify_works=yes ], [ - inotify_works=no + i_cv_inotify_works=no ]) ]) - if test $inotify_works = yes; then + if test $i_cv_inotify_works = yes; then have_notify=inotify notify=inotify AC_DEFINE(IOLOOP_NOTIFY_INOTIFY,, Use Linux inotify) @@ -809,7 +809,7 @@ dnl * make sure size_t isn't signed. we'd probably work fine with it, but dnl * it's more likely vulnerable to buffer overflows. Anyway, C99 specifies dnl * that it's unsigned and only some old systems define it as signed. -AC_CACHE_CHECK([whether size_t is signed],signed_size_t,[ +AC_CACHE_CHECK([whether size_t is signed],i_cv_signed_size_t,[ AC_RUN_IFELSE([AC_LANG_SOURCE([[ #include <sys/types.h> int main() { @@ -817,7 +817,7 @@ exit((size_t)(int)-1 <= 0 ? 0 : 1); } ]])],[ - signed_size_t=yes + i_cv_signed_size_t=yes echo echo "Your system's size_t is a signed integer, Dovecot isn't designed to" @@ -830,7 +830,7 @@ fi echo "..ignoring as requested.." ],[ - signed_size_t=no + i_cv_signed_size_t=no ],[]) ]) dnl Note: we check size_t rather than ssize_t here, because on OSX 10.2 @@ -924,7 +924,7 @@ AC_MSG_RESULT($i_cv_field_tm_gmtoff) dnl * how large time_t values does gmtime() accept? -AC_CACHE_CHECK([how large time_t values gmtime() accepts],gmtime_max_time_t,[ +AC_CACHE_CHECK([how large time_t values gmtime() accepts],i_cv_gmtime_max_time_t,[ AC_RUN_IFELSE([AC_LANG_SOURCE([[ #include <stdio.h> #include <time.h> @@ -955,16 +955,16 @@ return 0; } ]])],[ - gmtime_max_time_t=`cat conftest.temp` + i_cv_gmtime_max_time_t=`cat conftest.temp` rm -f conftest.temp ], [ printf "check failed, assuming " - gmtime_max_time_t=31 + i_cv_gmtime_max_time_t=31 ],[]) ]) -AC_DEFINE_UNQUOTED(TIME_T_MAX_BITS, $gmtime_max_time_t, max. time_t bits gmtime() can handle) +AC_DEFINE_UNQUOTED(TIME_T_MAX_BITS, $i_cv_gmtime_max_time_t, max. time_t bits gmtime() can handle) -AC_CACHE_CHECK([whether time_t is signed],signed_time_t,[ +AC_CACHE_CHECK([whether time_t is signed],i_cv_signed_time_t,[ AC_RUN_IFELSE([AC_LANG_SOURCE([[ #include <sys/types.h> int main() { @@ -972,12 +972,12 @@ exit((time_t)(int)-1 <= 0 ? 0 : 1); } ]])],[ - signed_time_t=yes + i_cv_signed_time_t=yes ], [ - signed_time_t=no + i_cv_signed_time_t=no ]) ]) -if test $signed_time_t = yes; then +if test $i_cv_signed_time_t = yes; then AC_DEFINE(TIME_T_SIGNED,, Define if your time_t is signed) fi @@ -1051,7 +1051,7 @@ ]) dnl * If mmap() plays nicely with write() -AC_CACHE_CHECK([whether shared mmaps get updated by write()s],mmap_plays_with_write,[ +AC_CACHE_CHECK([whether shared mmaps get updated by write()s],i_cv_mmap_plays_with_write,[ AC_TRY_RUN([ #include <stdio.h> #include <sys/types.h> @@ -1083,17 +1083,17 @@ return strcmp(mem, "3") == 0 ? 0 : 1; } ], [ - mmap_plays_with_write=yes + i_cv_mmap_plays_with_write=yes ], [ - mmap_plays_with_write=no + i_cv_mmap_plays_with_write=no ]) ]) -if test $mmap_plays_with_write = no; then +if test $i_cv_mmap_plays_with_write = no; then AC_DEFINE(MMAP_CONFLICTS_WRITE,, [Define if shared mmaps don't get updated by write()s]) fi dnl * see if fd passing works -AC_CACHE_CHECK([whether fd passing works],fd_passing,[ +AC_CACHE_CHECK([whether fd passing works],i_cv_fd_passing,[ for i in 1 2; do old_cflags="$CFLAGS" CFLAGS="$CFLAGS -I$srcdir/src/lib $srcdir/src/lib/fdpass.c" @@ -1143,20 +1143,20 @@ ], [ CFLAGS=$old_cflags if test $i = 2; then - fd_passing=buggy_cmsg_macros + i_cv_fd_passing=buggy_cmsg_macros else - fd_passing=yes + i_cv_fd_passing=yes fi break ], [ dnl no, try with BUGGY_CMSG_MACROS CFLAGS=$old_cflags - fd_passing=no + i_cv_fd_passing=no ]) done ]); -if test $fd_passing = buggy_cmsg_macros; then +if test $i_cv_fd_passing = buggy_cmsg_macros; then AC_DEFINE(BUGGY_CMSG_MACROS,, Define if you have buggy CMSG macros) fi @@ -1218,7 +1218,7 @@ AC_MSG_RESULT(no) ]) -AC_MSG_CHECKING([if struct stat has tv_nsec fields]) +AC_MSG_CHECKING([if struct stat has st_?tim timespec fields]) AC_TRY_COMPILE([ #include <sys/types.h> #include <sys/stat.h> @@ -1229,7 +1229,24 @@ return 0; ], [ - AC_DEFINE(HAVE_STAT_TV_NSEC,, Define if you have tv_nsec fields in struct stat) + AC_DEFINE(HAVE_STAT_XTIM,, Define if you have st_?tim timespec fields in struct stat) + AC_MSG_RESULT(yes) +], [ + AC_MSG_RESULT(no) +]) + +AC_MSG_CHECKING([if struct stat has st_?timespec fields]) +AC_TRY_COMPILE([ + #include <sys/types.h> + #include <sys/stat.h> + #include <unistd.h> +], [ + struct stat st; + unsigned long x = st.st_mtimespec.tv_nsec; + + return 0; +], [ + AC_DEFINE(HAVE_STAT_XTIMESPEC,, Define if you have st_?timespec fields in struct stat) AC_MSG_RESULT(yes) ], [ AC_MSG_RESULT(no) @@ -1335,7 +1352,7 @@ dnl *** C99 vsnprintf()? dnl *** -AC_CACHE_CHECK([for C99 vsnprintf()],c99_vsnprintf,[ +AC_CACHE_CHECK([for C99 vsnprintf()],i_cv_c99_vsnprintf,[ AC_RUN_IFELSE([AC_LANG_SOURCE([[ #include <stdio.h> #include <stdarg.h> @@ -1352,11 +1369,11 @@ int main() { return f("hello %s%d", "world", 1); }]])], - [c99_vsnprintf=yes], - [c99_vsnprintf=no], + [i_cv_c99_vsnprintf=yes], + [i_cv_c99_vsnprintf=no], []) ]) -if test $c99_vsnprintf = no; then +if test $i_cv_c99_vsnprintf = no; then AC_ERROR([You don't appear to have C99 compatible vsnprintf() call]) fi @@ -1616,29 +1633,44 @@ # we have a kludgy check here to check that we have # version >= v1.3. Although this doesn't work right with # non-MIT kerberos versioning.. - if `krb5-config --version|grep -v '1\.2' > /dev/null`; then - KRB5_LIBS=`krb5-config --libs gssapi` - KRB5_CFLAGS=`krb5-config --cflags gssapi` + if ! krb5-config --version gssapi 2>/dev/null > /dev/null; then + # krb5-config doesn't support gssapi. + KRB5_LIBS="`krb5-config --libs`" + KRB5_CFLAGS=`krb5-config --cflags` + AC_CHECK_LIB(gss, gss_acquire_cred, [ + # Solaris + KRB5_LIBS="$KRB5_LIBS -lgss" + ], [ + # failed + KRB5_LIBS= + ], $KRB5_LIBS) + elif krb5-config --version|grep '1\.2' > /dev/null; then + if test $want_gssapi = yes; then + AC_ERROR([Can't build with GSSAPI support: v1.2 library not supported]) + fi + else + KRB5_LIBS=`krb5-config --libs gssapi` + KRB5_CFLAGS=`krb5-config --cflags gssapi` + fi + if test "$KRB5_LIBS" != ""; then AC_SUBST(KRB5_LIBS) AC_SUBST(KRB5_CFLAGS) # Although krb5-config exists, all systems still don't # have gssapi.h old_CFLAGS=$CFLAGS - CFLAGS="$CFLAGS `krb5-config --cflags gssapi`" + CFLAGS="$CFLAGS $KRB5_CFLAGS" AC_CHECK_HEADER([gssapi/gssapi.h], [ AC_DEFINE(HAVE_GSSAPI_GSSAPI_H,, GSSAPI headers in gssapi/gssapi.h) have_gssapi=yes ]) - AC_CHECK_HEADER([gssapi/gssapi_ext.h], [ - AC_DEFINE(HAVE_GSSAPI_GSSAPI_EXT_H,, GSSAPI headers in gssapi/gssapi_ext.h) - ]) AC_CHECK_HEADER([gssapi.h], [ AC_DEFINE(HAVE_GSSAPI_H,, GSSAPI headers in gssapi.h) have_gssapi=yes ]) if test $have_gssapi = yes; then AC_DEFINE(HAVE_GSSAPI,, Build with GSSAPI support) + AC_CHECK_HEADERS(gssapi/gssapi_ext.h gssapi_krb5.h gssapi/gssapi_krb5.h) AC_CHECK_LIB(gss, __gss_userok, [ AC_DEFINE(HAVE___GSS_USEROK,, Define if you have __gss_userok()) @@ -1647,7 +1679,7 @@ if test x$want_gssapi_plugin != xyes; then AUTH_LIBS="$AUTH_LIBS $KRB5_LIBS" - AUTH_CFLAGS="$AUTH_CFLAGS `krb5-config --cflags gssapi`" + AUTH_CFLAGS="$AUTH_CFLAGS $KRB5_CFLAGS" AC_DEFINE(BUILTIN_GSSAPI,, GSSAPI support is built in) else have_gssapi_plugin=yes @@ -1658,10 +1690,6 @@ fi fi CFLAGS=$old_CFLAGS - else - if test $want_gssapi = yes; then - AC_ERROR([Can't build with GSSAPI support: v1.2 library not supported]) - fi fi else if test $want_gssapi = yes; then
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/dovecot-db.conf Mon Jun 09 05:11:18 2008 +0300 @@ -0,0 +1,11 @@ +# Example DB_CONFIG for Berkeley DB. Typically dict_db_config setting is used +# to point to this file. +# http://www.oracle.com/technology/documentation/berkeley-db/db/ref/env/db_config.html + +# Maximum number of simultaneous transactions. +set_tx_max 1000 + +# http://www.oracle.com/technology/documentation/berkeley-db/db/ref/lock/max.html +#set_lk_max_locks 1000 +#set_lk_max_lockers 1000 +#set_lk_max_objects 1000
--- a/dovecot-example.conf Sat May 17 17:50:54 2008 +0300 +++ b/dovecot-example.conf Mon Jun 09 05:11:18 2008 +0300 @@ -245,7 +245,7 @@ # There can be only one INBOX, and this setting defines which namespace # has it. - #inbox = yes + #inbox = no # If namespace is hidden, it's not advertised to clients via NAMESPACE # extension. You'll most likely also want to set list=no. This is mostly @@ -270,7 +270,7 @@ #mail_gid = # Group to enable temporarily for privileged operations. Currently this is -# used only for creating mbox dotlock files when creation fails for INBOX. +# used only with INBOX when either its initial creation or dotlocking fails. # Typically this is set to "mail" to give access to /var/mail. #mail_privileged_group = @@ -380,7 +380,8 @@ # specific users in user database by giving /./ in user's home directory # (eg. /home/./user chroots into /home). Note that usually there is no real # need to do chrooting, Dovecot doesn't allow users to access files outside -# their mail directory anyway. <doc/wiki/Chrooting.txt> +# their mail directory anyway. If your home directories are prefixed with +# the chroot directory, append "/." to mail_chroot. <doc/wiki/Chrooting.txt> #mail_chroot = ## @@ -1055,6 +1056,10 @@ #quota = mysql:/etc/dovecot-dict-quota.conf } +# Path to Berkeley DB's configuration file. See doc/dovecot-db.conf for an +# example. +#dict_db_config = + ## ## Plugin settings ##
--- a/src/auth/auth-master-listener.c Sat May 17 17:50:54 2008 +0300 +++ b/src/auth/auth-master-listener.c Mon Jun 09 05:11:18 2008 +0300 @@ -40,16 +40,16 @@ } static void -auth_master_listener_socket_free(struct auth_master_listener_socket *socket) +auth_master_listener_socket_free(struct auth_master_listener_socket *s) { - if (socket->path != NULL) { - (void)unlink(socket->path); - i_free(socket->path); + if (s->path != NULL) { + (void)unlink(s->path); + i_free(s->path); } - io_remove(&socket->io); - net_disconnect(socket->fd); - i_free(socket); + io_remove(&s->io); + net_disconnect(s->fd); + i_free(s); } void auth_master_listener_destroy(struct auth_master_listener *listener)
--- a/src/auth/auth-request.c Sat May 17 17:50:54 2008 +0300 +++ b/src/auth/auth-request.c Mon Jun 09 05:11:18 2008 +0300 @@ -870,11 +870,9 @@ static int is_ip_in_network(const char *network, const struct ip_addr *ip) { - const uint32_t *ip1, *ip2; struct ip_addr src_ip, net_ip; const char *p; - unsigned int max_bits, bits, pos, i; - uint32_t mask; + unsigned int max_bits, bits; if (net_ipv6_mapped_ipv4_convert(ip, &src_ip) == 0) ip = &src_ip; @@ -895,44 +893,7 @@ if (net_addr2ip(network, &net_ip) < 0) return -1; - if (IPADDR_IS_V4(ip) != IPADDR_IS_V4(&net_ip)) { - /* one is IPv6 and one is IPv4 */ - return 0; - } - i_assert(IPADDR_IS_V6(ip) == IPADDR_IS_V6(&net_ip)); - - if (IPADDR_IS_V4(ip)) { - ip1 = &ip->u.ip4.s_addr; - ip2 = &net_ip.u.ip4.s_addr; - } else { -#ifdef HAVE_IPV6 - ip1 = (const void *)&ip->u.ip6; - ip2 = (const void *)&net_ip.u.ip6; -#else - /* shouldn't get here */ - return -1; -#endif - } - - /* check first the full 32bit ints */ - for (pos = 0, i = 0; pos + 32 <= bits; pos += 32, i++) { - if (ip1[i] != ip2[i]) - return 0; - } - - /* check the last full bytes */ - for (mask = 0xff; pos + 8 <= bits; pos += 8, mask <<= 8) { - if ((ip1[i] & mask) != (ip2[i] & mask)) - return 0; - } - - /* check the last bits, they're reversed in bytes */ - bits -= pos; - for (mask = 0x80 << (pos % 32); bits > 0; bits--, mask >>= 1) { - if ((ip1[i] & mask) != (ip2[i] & mask)) - return 0; - } - return 1; + return net_is_in_network(ip, &net_ip, bits); } static void auth_request_validate_networks(struct auth_request *request,
--- a/src/auth/auth-stream.c Sat May 17 17:50:54 2008 +0300 +++ b/src/auth/auth-stream.c Mon Jun 09 05:11:18 2008 +0300 @@ -15,7 +15,7 @@ struct auth_stream_reply *reply; reply = p_new(pool, struct auth_stream_reply, 1); - reply->str = str_new(pool, 256); + reply->str = str_new(pool, 128); return reply; }
--- a/src/auth/db-ldap.c Sat May 17 17:50:54 2008 +0300 +++ b/src/auth/db-ldap.c Mon Jun 09 05:11:18 2008 +0300 @@ -979,8 +979,15 @@ ctx->attr_map = attr_map; static_data = hash_lookup(attr_map, ""); - if (static_data != NULL) - ctx->static_attrs = t_strsplit(static_data, ","); + if (static_data != NULL) { + const struct var_expand_table *table; + string_t *str; + + table = auth_request_get_var_expand_table(auth_request, NULL); + str = t_str_new(256); + var_expand(str, static_data, table); + ctx->static_attrs = t_strsplit(str_c(str), ","); + } if (auth_request->auth->verbose_debug) ctx->debug = t_str_new(256); @@ -1009,6 +1016,7 @@ db_ldap_result_change_attr(struct db_ldap_result_iterate_context *ctx) { ctx->name = hash_lookup(ctx->attr_map, ctx->attr); + ctx->template = NULL; if (ctx->debug != NULL) { str_printfa(ctx->debug, " %s(%s)=", ctx->attr, @@ -1095,6 +1103,9 @@ ctx->name = t_strdup_until(*ctx->static_attrs, p); ctx->value = p + 1; } + /* make _next_all() return correct values */ + ctx->template = ""; + ctx->val_1_arr[0] = ctx->value; ctx->static_attrs++; return TRUE; }
--- a/src/auth/db-passwd-file.c Sat May 17 17:50:54 2008 +0300 +++ b/src/auth/db-passwd-file.c Mon Jun 09 05:11:18 2008 +0300 @@ -178,7 +178,7 @@ pw->stamp = st.st_mtime; pw->size = st.st_size; - pw->pool = pool_alloconly_create("passwd_file", 10240);; + pw->pool = pool_alloconly_create(MEMPOOL_GROWING"passwd_file", 10240); pw->users = hash_create(default_pool, pw->pool, 100, str_hash, (hash_cmp_callback_t *)strcmp); @@ -385,15 +385,12 @@ struct passwd_file *pw; struct passwd_user *pu; const struct var_expand_table *table; - string_t *username; + string_t *username, *dest; const char *path; if (!db->vars) pw = db->default_file; else { - const struct var_expand_table *table; - string_t *dest; - table = auth_request_get_var_expand_table(request, path_fix); dest = t_str_new(256); var_expand(dest, db->path, table);
--- a/src/auth/mech-gssapi.c Sat May 17 17:50:54 2008 +0300 +++ b/src/auth/mech-gssapi.c Mon Jun 09 05:11:18 2008 +0300 @@ -23,12 +23,25 @@ #ifdef HAVE_GSSAPI +#ifndef HAVE___GSS_USEROK +# define USE_KRB5_USEROK +# include <krb5.h> +#endif + #ifdef HAVE_GSSAPI_GSSAPI_H # include <gssapi/gssapi.h> #elif defined (HAVE_GSSAPI_H) # include <gssapi.h> #endif +#ifdef HAVE_GSSAPI_GSSAPI_KRB5_H +# include <gssapi/gssapi_krb5.h> +#elif defined (HAVE_GSSAPI_KRB5_H) +# include <gssapi_krb5.h> +#else +# undef USE_KRB5_USEROK +#endif + #ifdef HAVE_GSSAPI_GSSAPI_EXT_H # include <gssapi/gssapi_ext.h> #endif @@ -156,6 +169,7 @@ return major_status; } +#ifndef HAVE___GSS_USEROK static gss_name_t import_name(struct auth_request *request, void *str, size_t len) { @@ -177,6 +191,7 @@ return name; } +#endif static void gssapi_sec_context(struct gssapi_auth_request *request, gss_buffer_desc inbuf) @@ -271,6 +286,61 @@ request->sasl_gssapi_state = GSS_STATE_UNWRAP; } +#ifdef USE_KRB5_USEROK +static bool gssapi_krb5_userok(struct gssapi_auth_request *request) +{ + krb5_context ctx; + krb5_principal princ; + krb5_error_code krb5_err; + OM_uint32 major_status, minor_status; + gss_buffer_desc princ_name; + gss_OID name_type; + const char *princ_display_name; + bool ret = FALSE; + + /* Parse out the principal's username */ + major_status = gss_display_name(&minor_status, request->authn_name, + &princ_name, &name_type); + if (major_status != GSS_S_COMPLETE) { + auth_request_log_gss_error(&request->auth_request, major_status, + GSS_C_GSS_CODE, + "gssapi_krb5_userok"); + return FALSE; + } + if (name_type != GSS_KRB5_NT_PRINCIPAL_NAME) { + auth_request_log_error(&request->auth_request, "gssapi", + "OID not kerberos principal name"); + return FALSE; + } + princ_display_name = t_strndup(princ_name.value, princ_name.length); + gss_release_buffer(&minor_status, &princ_name); + + /* Init a krb5 context and parse the principal username */ + krb5_err = krb5_init_context(&ctx); + if (krb5_err != 0) { + auth_request_log_error(&request->auth_request, "gssapi", + "krb5_init_context() failed: %d", (int)krb5_err); + return FALSE; + } + krb5_err = krb5_parse_name(ctx, princ_display_name, &princ); + if (krb5_err != 0) { + /* writing the error string would be better, but we probably + rarely get here and there doesn't seem to be a standard + way of getting it */ + auth_request_log_error(&request->auth_request, "gssapi", + "krb5_parse_name() failed: %d", + (int)krb5_err); + } else { + /* See if the principal is authorized to act as the + specified user */ + ret = krb5_kuserok(ctx, princ, request->auth_request.user); + krb5_free_principal(ctx, princ); + } + krb5_free_context(ctx); + return ret; +} +#endif + static void gssapi_unwrap(struct gssapi_auth_request *request, gss_buffer_desc inbuf) { @@ -334,22 +404,26 @@ auth_request_fail(&request->auth_request); return; } + + request->auth_request.user = + p_strndup(request->auth_request.pool, + (unsigned char *)outbuf.value + 4, + outbuf.length - 4); + major_status = gss_compare_name(&minor_status, request->authn_name, request->authz_name, &equal_authn_authz); +#ifdef USE_KRB5_USEROK + if (equal_authn_authz == 0) + equal_authn_authz = gssapi_krb5_userok(request); +#endif if (equal_authn_authz == 0) { auth_request_log_error(&request->auth_request, "gssapi", "authn_name and authz_name differ: not supported"); auth_request_fail(&request->auth_request); return; } - - request->auth_request.user = - p_strndup(request->auth_request.pool, - (unsigned char *)outbuf.value + 4, - outbuf.length - 4); - #endif auth_request_success(&request->auth_request, NULL, 0); }
--- a/src/auth/passdb.c Sat May 17 17:50:54 2008 +0300 +++ b/src/auth/passdb.c Mon Jun 09 05:11:18 2008 +0300 @@ -78,9 +78,15 @@ if (!password_scheme_is_alias(input_scheme, wanted_scheme)) { if (!password_scheme_is_alias(input_scheme, "PLAIN")) { - auth_request_log_info(auth_request, "password", + const char *error = t_strdup_printf( "Requested %s scheme, but we have only %s", wanted_scheme, input_scheme); + if (auth_request->auth->verbose_debug_passwords) { + error = t_strdup_printf("%s (input: %s)", + error, input); + } + auth_request_log_info(auth_request, "password", + "%s", error); return FALSE; }
--- a/src/auth/userdb-nss.c Sat May 17 17:50:54 2008 +0300 +++ b/src/auth/userdb-nss.c Mon Jun 09 05:11:18 2008 +0300 @@ -130,13 +130,13 @@ static void userdb_nss_deinit(struct userdb_module *_module) { struct nss_userdb_module *module = (struct nss_userdb_module *)_module; - void (*endpwent)(void); + void (*mod_endpwent)(void); + const char *symbol; - endpwent = module_get_symbol(&module->nss_module, - t_strdup_printf("_nss_%s_endpwent", - module->nss_module.name)); - if (endpwent != NULL) - endpwent(); + symbol = t_strdup_printf("_nss_%s_endpwent", module->nss_module.name); + mod_endpwent = module_get_symbol(&module->nss_module, symbol); + if (mod_endpwent != NULL) + mod_endpwent(); } struct userdb_module_interface userdb_nss = {
--- a/src/deliver/auth-client.c Sat May 17 17:50:54 2008 +0300 +++ b/src/deliver/auth-client.c Mon Jun 09 05:11:18 2008 +0300 @@ -95,8 +95,10 @@ const char *const *tmp, *extra_groups; uid_t uid = 0; gid_t gid = 0; - const char *chroot = getenv("MAIL_CHROOT"); + const char *chroot_dir = getenv("MAIL_CHROOT"); + const char *home_dir = NULL; bool debug = getenv("DEBUG") != NULL; + unsigned int len; for (tmp = t_strsplit(args, "\t"); *tmp != NULL; tmp++) { if (debug) @@ -110,10 +112,6 @@ conn->user); return_value = EX_TEMPFAIL; } - if (conn->euid != uid) { - env_put(t_strconcat("RESTRICT_SETUID=", - dec2str(uid), NULL)); - } } else if (strncmp(*tmp, "gid=", 4) == 0) { gid = strtoul(*tmp + 4, NULL, 10); @@ -122,24 +120,19 @@ conn->user); return_value = EX_TEMPFAIL; } - - if (conn->euid == 0 || getegid() != gid) { - env_put(t_strconcat("RESTRICT_SETGID=", - *tmp + 4, NULL)); - } } else if (strncmp(*tmp, "chroot=", 7) == 0) { - chroot = *tmp + 7; + chroot_dir = *tmp + 7; } else { char *field = i_strdup(*tmp); if (strncmp(field, "home=", 5) == 0) - env_put(t_strconcat("HOME=", field + 5, NULL)); + home_dir = field + 5; array_append(conn->extra_fields, &field, 1); } } - if (uid == 0 && getenv("MAIL_UID")) { + if (uid == 0 && getenv("MAIL_UID") != NULL) { if (!parse_uid(getenv("MAIL_UID"), &uid) || uid == 0) { i_error("mail_uid setting is invalid"); return_value = EX_TEMPFAIL; @@ -151,7 +144,7 @@ return_value = EX_TEMPFAIL; return; } - if (gid == 0 && getenv("MAIL_GID")) { + if (gid == 0 && getenv("MAIL_GID") != NULL) { if (!parse_gid(getenv("MAIL_GID"), &gid) || gid == 0) { i_error("mail_gid setting is invalid"); return_value = EX_TEMPFAIL; @@ -164,8 +157,23 @@ return; } - if (chroot != NULL) - env_put(t_strconcat("RESTRICT_CHROOT=", chroot, NULL)); + if (conn->euid != uid) + env_put(t_strconcat("RESTRICT_SETUID=", dec2str(uid), NULL)); + if (conn->euid == 0 || getegid() != gid) + env_put(t_strconcat("RESTRICT_SETGID=", dec2str(gid), NULL)); + + if (chroot_dir != NULL) { + len = strlen(chroot_dir); + if (len > 2 && strcmp(chroot_dir + len - 2, "/.") == 0 && + home_dir != NULL && + strncmp(home_dir, chroot_dir, len - 2) == 0) { + /* strip chroot dir from home dir */ + home_dir += len - 2; + } + env_put(t_strconcat("RESTRICT_CHROOT=", chroot_dir, NULL)); + } + if (home_dir != NULL) + env_put(t_strconcat("HOME=", home_dir, NULL)); extra_groups = getenv("MAIL_EXTRA_GROUPS"); if (extra_groups != NULL) {
--- a/src/deliver/deliver.c Sat May 17 17:50:54 2008 +0300 +++ b/src/deliver/deliver.c Mon Jun 09 05:11:18 2008 +0300 @@ -633,7 +633,7 @@ static void print_help(void) { printf( -"Usage: deliver [-c <config file>] [-a <address>] [-d <username>]\n" +"Usage: deliver [-c <config file>] [-a <address>] [-d <username>] [-p <path>]\n" " [-f <envelope sender>] [-m <mailbox>] [-n] [-e] [-k]\n"); } @@ -711,7 +711,7 @@ const char *config_path = DEFAULT_CONFIG_FILE; const char *mailbox = "INBOX"; const char *auth_socket; - const char *home, *destaddr, *user, *value, *error; + const char *home, *destaddr, *user, *value, *errstr, *path; ARRAY_TYPE(string) extra_fields; struct mail_namespace *ns, *raw_ns; struct mail_storage *storage; @@ -743,7 +743,7 @@ lib_signals_ignore(SIGXFSZ, TRUE); #endif - destaddr = user = NULL; + destaddr = user = path = NULL; for (i = 1; i < argc; i++) { if (strcmp(argv[i], "-a") == 0) { /* destination address */ @@ -758,6 +758,12 @@ i_fatal_status(EX_USAGE, "Missing -d argument"); user = argv[i]; user_auth = TRUE; + } else if (strcmp(argv[i], "-p") == 0) { + /* input path */ + i++; + if (i == argc) + i_fatal_status(EX_USAGE, "Missing -p argument"); + path = argv[i]; } else if (strcmp(argv[i], "-e") == 0) { stderr_rejection = TRUE; } else if (strcmp(argv[i], "-c") == 0) { @@ -938,11 +944,19 @@ raw_ns = mail_namespaces_init_empty(namespace_pool); raw_ns->flags |= NAMESPACE_FLAG_INTERNAL; if (mail_storage_create(raw_ns, "raw", "/tmp", user, - 0, FILE_LOCK_METHOD_FCNTL, &error) < 0) - i_fatal("Couldn't create internal raw storage: %s", error); - input = create_raw_stream(0, &mtime); - box = mailbox_open(raw_ns->storage, "Dovecot Delivery Mail", input, - MAILBOX_OPEN_NO_INDEX_FILES); + MAIL_STORAGE_FLAG_FULL_FS_ACCESS, + FILE_LOCK_METHOD_FCNTL, &errstr) < 0) + i_fatal("Couldn't create internal raw storage: %s", errstr); + if (path == NULL) { + input = create_raw_stream(0, &mtime); + box = mailbox_open(raw_ns->storage, "Dovecot Delivery Mail", + input, MAILBOX_OPEN_NO_INDEX_FILES); + i_stream_unref(&input); + } else { + mtime = (time_t)-1; + box = mailbox_open(raw_ns->storage, path, NULL, + MAILBOX_OPEN_NO_INDEX_FILES); + } if (box == NULL) i_fatal("Can't open delivery mail as raw"); if (mailbox_sync(box, 0, 0, NULL) < 0) { @@ -978,20 +992,17 @@ if (ret < 0 && !tried_default_save) { /* plugins didn't handle this. save into the default mailbox. */ - i_stream_seek(input, 0); ret = deliver_save(ns, &storage, mailbox, mail, 0, NULL); } if (ret < 0 && strcasecmp(mailbox, "INBOX") != 0) { /* still didn't work. try once more to save it to INBOX. */ - i_stream_seek(input, 0); ret = deliver_save(ns, &storage, "INBOX", mail, 0, NULL); } if (ret < 0 ) { const char *error_string; enum mail_error error; - int ret; if (storage == NULL) { /* This shouldn't happen */ @@ -1000,6 +1011,13 @@ } error_string = mail_storage_get_last_error(storage, &error); + + if (stderr_rejection) { + /* write to stderr also for tempfails so that MTA + can log the reason if it wants to. */ + fprintf(stderr, "%s\n", error_string); + } + if (error != MAIL_ERROR_NOSPACE || getenv("QUOTA_FULL_TEMPFAIL") != NULL) { /* Saving to INBOX should always work unless @@ -1008,20 +1026,17 @@ return EX_TEMPFAIL; } + /* we'll have to reply with permanent failure */ deliver_log(mail, "rejected: %s", str_sanitize(error_string, 512)); - /* we'll have to reply with permanent failure */ - if (stderr_rejection) { - fprintf(stderr, "%s\n", error_string); + if (stderr_rejection) return EX_NOPERM; - } ret = mail_send_rejection(mail, user, error_string); if (ret != 0) return ret < 0 ? EX_TEMPFAIL : ret; /* ok, rejection sent */ } - i_stream_unref(&input); i_free(explicit_envelope_sender); mail_free(&mail);
--- a/src/deliver/duplicate.c Sat May 17 17:50:54 2008 +0300 +++ b/src/deliver/duplicate.c Mon Jun 09 05:11:18 2008 +0300 @@ -245,7 +245,7 @@ } void duplicate_mark(const void *id, size_t id_size, - const char *user, time_t time) + const char *user, time_t timestamp) { struct duplicate *d; void *new_id; @@ -260,7 +260,7 @@ d->id = new_id; d->id_size = id_size; d->user = p_strdup(duplicate_file->pool, user); - d->time = time; + d->time = timestamp; duplicate_file->changed = TRUE; hash_insert(duplicate_file->hash, d, d);
--- a/src/deliver/duplicate.h Sat May 17 17:50:54 2008 +0300 +++ b/src/deliver/duplicate.h Mon Jun 09 05:11:18 2008 +0300 @@ -5,7 +5,7 @@ int duplicate_check(const void *id, size_t id_size, const char *user); void duplicate_mark(const void *id, size_t id_size, - const char *user, time_t time); + const char *user, time_t timestamp); void duplicate_flush(void);
--- a/src/imap-login/client-authenticate.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap-login/client-authenticate.c Mon Jun 09 05:11:18 2008 +0300 @@ -156,7 +156,7 @@ } client_send_tagline(client, str_c(reply)); if (!nologin) { - client_destroy(client, "Login with referral"); + client_destroy_success(client, "Login with referral"); return TRUE; } } else if (nologin || proxy_self) { @@ -209,7 +209,7 @@ } client_send_tagline(client, "OK Logged in."); - client_destroy(client, "Login"); + client_destroy_success(client, "Login"); break; case SASL_SERVER_REPLY_AUTH_FAILED: case SASL_SERVER_REPLY_CLIENT_ERROR: @@ -234,7 +234,9 @@ else { client_send_tagline(client, t_strconcat("NO ", data, NULL)); - client_destroy(client, data); + /* authentication itself succeeded, we just hit some + internal failure. */ + client_destroy_success(client, data); } break; case SASL_SERVER_REPLY_CONTINUE:
--- a/src/imap-login/client.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap-login/client.c Mon Jun 09 05:11:18 2008 +0300 @@ -211,9 +211,7 @@ client_destroy(client, "Aborted login " "(tried to use disabled plaintext authentication)"); } else { - client_destroy(client, t_strdup_printf( - "Aborted login (%u authentication attempts)", - client->common.auth_attempts)); + client_destroy(client, "Aborted login"); } return 1; } @@ -284,8 +282,8 @@ if (fatal) { client_send_line(client, t_strconcat("* BYE ", msg, NULL)); - client_destroy(client, t_strconcat("Disconnected: ", - msg, NULL)); + client_destroy(client, + t_strconcat("Disconnected: ", msg, NULL)); return FALSE; } @@ -311,8 +309,8 @@ if (++client->bad_counter >= CLIENT_MAX_BAD_COMMANDS) { client_send_line(client, "* BYE Too many invalid IMAP commands."); - client_destroy(client, "Disconnected: " - "Too many invalid commands"); + client_destroy(client, + "Disconnected: Too many invalid commands"); return FALSE; } client_send_tagline(client, @@ -486,6 +484,10 @@ return; client->destroyed = TRUE; + if (!client->login_success && reason != NULL) { + reason = t_strdup_printf("%s (auth failed, %u attempts)", + reason, client->common.auth_attempts); + } if (reason != NULL) client_syslog(&client->common, reason); @@ -543,6 +545,12 @@ main_unref(); } +void client_destroy_success(struct imap_client *client, const char *reason) +{ + client->login_success = TRUE; + client_destroy(client, reason); +} + void client_destroy_internal_failure(struct imap_client *client) { client_send_line(client, "* BYE Internal login failure. "
--- a/src/imap-login/client.h Sat May 17 17:50:54 2008 +0300 +++ b/src/imap-login/client.h Mon Jun 09 05:11:18 2008 +0300 @@ -24,6 +24,7 @@ const char *cmd_tag, *cmd_name; + unsigned int login_success:1; unsigned int cmd_finished:1; unsigned int proxy_login_sent:1; unsigned int skip_line:1; @@ -33,6 +34,7 @@ }; void client_destroy(struct imap_client *client, const char *reason); +void client_destroy_success(struct imap_client *client, const char *reason); void client_destroy_internal_failure(struct imap_client *client); void client_send_line(struct imap_client *client, const char *line);
--- a/src/imap-login/imap-proxy.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap-login/imap-proxy.c Mon Jun 09 05:11:18 2008 +0300 @@ -46,9 +46,12 @@ return 0; } else if (strncmp(line, "P OK ", 5) == 0) { /* Login successful. Send this line to client. */ - (void)o_stream_send_str(client->output, client->cmd_tag); - (void)o_stream_send_str(client->output, line + 1); - (void)o_stream_send(client->output, "\r\n", 2); + str = t_str_new(128); + str_append(str, client->cmd_tag); + str_append(str, line + 1); + str_append(str, "\r\n"); + (void)o_stream_send(client->output, + str_data(str), str_len(str)); msg = t_strdup_printf("proxy(%s): started proxying to %s:%u", client->common.virtual_user, @@ -63,7 +66,7 @@ client->input = NULL; client->output = NULL; client->common.fd = -1; - client_destroy(client, msg); + client_destroy_success(client, msg); return -1; } else if (strncmp(line, "P ", 2) == 0) { /* If the backend server isn't Dovecot, the error message may @@ -117,7 +120,7 @@ /* failed for some reason, probably server disconnected */ client_send_line(client, "* BYE Temporary login failure."); - client_destroy(client, NULL); + client_destroy_success(client, NULL); return; } @@ -132,7 +135,7 @@ return; case -1: /* disconnected */ - client_destroy(client, "Proxy: Remote disconnected"); + client_destroy_success(client, "Proxy: Remote disconnected"); return; }
--- a/src/imap/client.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/client.c Mon Jun 09 05:11:18 2008 +0300 @@ -66,17 +66,36 @@ return client; } -void client_command_cancel(struct client_command_context *cmd) +void client_command_cancel(struct client_command_context **_cmd) { + struct client_command_context *cmd = *_cmd; bool cmd_ret; - cmd->cancel = TRUE; - cmd_ret = cmd->func == NULL ? TRUE : cmd->func(cmd); + switch (cmd->state) { + case CLIENT_COMMAND_STATE_WAIT_INPUT: + /* a bit kludgy check: cancel command only if it has context + set. currently only append command matches this check. all + other commands haven't even started the processing yet. */ + if (cmd->context == NULL) + break; + /* fall through */ + case CLIENT_COMMAND_STATE_WAIT_OUTPUT: + cmd->cancel = TRUE; + break; + case CLIENT_COMMAND_STATE_WAIT_UNAMBIGUITY: + case CLIENT_COMMAND_STATE_WAIT_SYNC: + /* commands haven't started yet */ + break; + case CLIENT_COMMAND_STATE_DONE: + i_unreached(); + } + + cmd_ret = !cmd->cancel || cmd->func == NULL ? TRUE : cmd->func(cmd); if (!cmd_ret && cmd->state != CLIENT_COMMAND_STATE_DONE) { if (cmd->client->output->closed) i_panic("command didn't cancel itself: %s", cmd->name); } else { - client_command_free(cmd); + client_command_free(_cmd); } } @@ -112,6 +131,7 @@ void client_destroy(struct client *client, const char *reason) { + struct client_command_context *cmd; i_assert(!client->destroyed); client->destroyed = TRUE; @@ -127,11 +147,17 @@ /* finish off all the queued commands. */ if (client->output_lock != NULL) - client_command_cancel(client->output_lock); + client_command_cancel(&client->output_lock); + while (client->command_queue != NULL) { + cmd = client->command_queue; + client_command_cancel(&cmd); + } + /* handle the input_lock command last. it might have been waiting on + other queued commands (although we probably should just drop the + command at that point since it hasn't started running. but this may + change in future). */ if (client->input_lock != NULL) - client_command_cancel(client->input_lock); - while (client->command_queue != NULL) - client_command_cancel(client->command_queue); + client_command_cancel(&client->input_lock); if (client->mailbox != NULL) { client_search_updates_free(client); @@ -354,7 +380,12 @@ enum command_flags flags; bool broken_client = FALSE; - if ((cmd->cmd_flags & COMMAND_FLAG_USES_SEQS) != 0) { + if ((cmd->cmd_flags & COMMAND_FLAG_BREAKS_MAILBOX) == + COMMAND_FLAG_BREAKS_MAILBOX) { + /* there must be no other command running that uses the + selected mailbox */ + flags = COMMAND_FLAG_USES_MAILBOX; + } else if ((cmd->cmd_flags & COMMAND_FLAG_USES_SEQS) != 0) { /* no existing command must be breaking sequences */ flags = COMMAND_FLAG_BREAKS_SEQS; broken_client = TRUE; @@ -409,10 +440,13 @@ return cmd; } -void client_command_free(struct client_command_context *cmd) +void client_command_free(struct client_command_context **_cmd) { + struct client_command_context *cmd = *_cmd; struct client *client = cmd->client; + *_cmd = NULL; + /* reset input idle time because command output might have taken a long time and we don't want to disconnect client immediately then */ client->last_input = ioloop_time; @@ -556,7 +590,7 @@ /* command is being executed - continue it */ if (cmd->func(cmd) || cmd->state == CLIENT_COMMAND_STATE_DONE) { /* command execution was finished */ - client_command_free(cmd); + client_command_free(&cmd); client_add_missing_io(client); return TRUE; } @@ -598,7 +632,7 @@ /* unknown command */ client_send_command_error(cmd, "Unknown command."); cmd->param_error = TRUE; - client_command_free(cmd); + client_command_free(&cmd); return TRUE; } else { i_assert(!client->disconnected); @@ -709,7 +743,7 @@ client_command_new(client); cmd->param_error = TRUE; client_send_command_error(cmd, "Too long argument."); - client_command_free(cmd); + client_command_free(&cmd); } o_stream_uncork(output); o_stream_unref(&output); @@ -726,7 +760,7 @@ (void)client_handle_unfinished_cmd(cmd); else { /* command execution was finished */ - client_command_free(cmd); + client_command_free(&cmd); } }
--- a/src/imap/client.h Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/client.h Mon Jun 09 05:11:18 2008 +0300 @@ -158,8 +158,8 @@ void clients_init(void); void clients_deinit(void); -void client_command_cancel(struct client_command_context *cmd); -void client_command_free(struct client_command_context *cmd); +void client_command_cancel(struct client_command_context **cmd); +void client_command_free(struct client_command_context **cmd); bool client_handle_unfinished_cmd(struct client_command_context *cmd); void client_continue_pending_input(struct client **_client);
--- a/src/imap/cmd-append.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/cmd-append.c Mon Jun 09 05:11:18 2008 +0300 @@ -51,7 +51,7 @@ cmd_append_finish(cmd->context); /* Reset command so that client_destroy() doesn't try to call cmd_append_continue_message() anymore. */ - client_command_free(cmd); + client_command_free(&cmd); client_destroy(client, "Disconnected in APPEND"); return; case -2: @@ -69,7 +69,7 @@ client_send_command_error(cmd, "Too long argument."); cmd->param_error = TRUE; - client_command_free(cmd); + client_command_free(&cmd); return; } @@ -79,7 +79,7 @@ if (!finished && cmd->state != CLIENT_COMMAND_STATE_DONE) (void)client_handle_unfinished_cmd(cmd); else - client_command_free(cmd); + client_command_free(&cmd); (void)cmd_sync_delayed(client); client_continue_pending_input(&client); }
--- a/src/imap/cmd-idle.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/cmd-idle.c Mon Jun 09 05:11:18 2008 +0300 @@ -54,7 +54,7 @@ o_stream_uncork(client->output); if (free_cmd) - client_command_free(ctx->cmd); + client_command_free(&ctx->cmd); } static void idle_client_input(struct cmd_idle_context *ctx) @@ -102,12 +102,13 @@ return; } - /* Sending this keeps NATs/stateful firewalls alive, and it also - updates client->last_output so we don't ever disconnect the - client. Sending this output should kill dead connections and there - are several clients that really want to IDLE forever (Outlook - especially). */ + /* Sending this keeps NATs/stateful firewalls alive. Sending this + also catches dead connections. */ client_send_line(ctx->client, "* OK Still here"); + /* Make sure idling connections don't get disconnected. There are + several clients that really want to IDLE forever and there's not + much harm in letting them do so. */ + timeout_reset(ctx->client->to_idle); } static void idle_sync_now(struct mailbox *box, struct cmd_idle_context *ctx)
--- a/src/imap/cmd-list.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/cmd-list.c Mon Jun 09 05:11:18 2008 +0300 @@ -85,7 +85,7 @@ return; if ((flags & MAILBOX_CHILD_SUBSCRIBED) != 0) - str_append(str, " (\"CHILDINFO\" (\"SUBSCRIBED\"))"); + str_append(str, " (CHILDINFO (\"SUBSCRIBED\"))"); } static bool @@ -347,14 +347,14 @@ imap_quote_append_string(str, name, FALSE); mailbox_childinfo2str(ctx, str, flags); + ret = client_send_line(ctx->cmd->client, str_c(str)); if (ctx->status_items != 0 && (flags & (MAILBOX_NONEXISTENT | MAILBOX_NOSELECT)) == 0) { T_BEGIN { list_send_status(ctx, name); } T_END; } - - if (client_send_line(ctx->cmd->client, str_c(str)) == 0) { + if (ret == 0) { /* buffer is full, continue later */ return 0; } @@ -812,7 +812,7 @@ args += 2; } - ctx->list_flags = MAILBOX_LIST_ITER_VIRTUAL_NAMES; + ctx->list_flags |= MAILBOX_LIST_ITER_VIRTUAL_NAMES; if (lsub) { /* LSUB - we don't care about flags */ ctx->list_flags |= MAILBOX_LIST_ITER_SELECT_SUBSCRIBED |
--- a/src/imap/cmd-search.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/cmd-search.c Mon Jun 09 05:11:18 2008 +0300 @@ -465,7 +465,7 @@ if (!finished) (void)client_handle_unfinished_cmd(cmd); else - client_command_free(cmd); + client_command_free(&cmd); (void)cmd_sync_delayed(client); client_continue_pending_input(&client); }
--- a/src/imap/cmd-store.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/cmd-store.c Mon Jun 09 05:11:18 2008 +0300 @@ -125,7 +125,7 @@ ARRAY_TYPE(seq_range) modified_set = ARRAY_INIT; enum mailbox_transaction_flags flags = 0; enum imap_sync_flags imap_sync_flags = 0; - const char *tagged_reply; + const char *reply, *tagged_reply; string_t *str; int ret; @@ -149,10 +149,21 @@ if (!store_parse_args(&ctx, ++args)) return TRUE; + if (mailbox_is_readonly(client->mailbox)) { + if (ctx.max_modseq < (uint64_t)-1) + reply = "NO CONDSTORE failed: Mailbox is read-only."; + else + reply = "OK Store ignored with read-only mailbox."; + return cmd_sync(cmd, MAILBOX_SYNC_FLAG_FAST | + (cmd->uid ? 0 : MAILBOX_SYNC_FLAG_NO_EXPUNGES), + 0, reply); + } + if (ctx.silent) flags |= MAILBOX_TRANSACTION_FLAG_HIDE; if (ctx.max_modseq < (uint64_t)-1) flags |= MAILBOX_TRANSACTION_FLAG_REFRESH; + t = mailbox_transaction_begin(client->mailbox, flags); search_ctx = mailbox_search_init(t, search_args, NULL); mail_search_args_unref(&search_args);
--- a/src/imap/cmd-subscribe.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/cmd-subscribe.c Mon Jun 09 05:11:18 2008 +0300 @@ -28,8 +28,6 @@ bool cmd_subscribe_full(struct client_command_context *cmd, bool subscribe) { struct mail_namespace *ns; - struct mail_storage *storage; - struct mailbox_list *list; const char *mailbox, *verify_name; /* <mailbox> */ @@ -43,11 +41,10 @@ client_send_tagline(cmd, "NO Unknown namespace."); return TRUE; } - storage = ns->storage; if ((client_workarounds & WORKAROUND_TB_EXTRA_MAILBOX_SEP) != 0 && *mailbox != '\0' && mailbox[strlen(mailbox)-1] == - mail_storage_get_hierarchy_sep(storage)) { + mail_storage_get_hierarchy_sep(ns->storage)) { /* verify the validity without the trailing '/' */ verify_name = t_strndup(verify_name, strlen(verify_name)-1); } @@ -61,9 +58,8 @@ return TRUE; } - list = mail_storage_get_list(storage); - if (mailbox_list_set_subscribed(list, mailbox, subscribe) < 0) - client_send_storage_error(cmd, storage); + if (mailbox_list_set_subscribed(ns->list, mailbox, subscribe) < 0) + client_send_list_error(cmd, ns->list); else { client_send_tagline(cmd, subscribe ? "OK Subscribe completed." :
--- a/src/imap/cmd-x-cancel.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/cmd-x-cancel.c Mon Jun 09 05:11:18 2008 +0300 @@ -16,7 +16,7 @@ for (; cancel_cmd != NULL; cancel_cmd = cancel_cmd->next) { if (cancel_cmd->tag != NULL && cancel_cmd != cmd && strcmp(cancel_cmd->tag, tag) == 0) { - client_command_cancel(cancel_cmd); + client_command_cancel(&cancel_cmd); client_send_tagline(cmd, "OK Command cancelled."); return TRUE; }
--- a/src/imap/commands.h Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/commands.h Mon Jun 09 05:11:18 2008 +0300 @@ -16,7 +16,11 @@ /* Command may reply with EXPUNGE, causing sequences to break */ COMMAND_FLAG_BREAKS_SEQS = 0x02, /* Command changes the mailbox */ - COMMAND_FLAG_BREAKS_MAILBOX = 0x04 | COMMAND_FLAG_BREAKS_SEQS + COMMAND_FLAG_BREAKS_MAILBOX = 0x04 | COMMAND_FLAG_BREAKS_SEQS, + + /* Command uses selected mailbox */ + COMMAND_FLAG_USES_MAILBOX = COMMAND_FLAG_BREAKS_MAILBOX | + COMMAND_FLAG_USES_SEQS }; struct command {
--- a/src/imap/imap-fetch-body.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/imap-fetch-body.c Mon Jun 09 05:11:18 2008 +0300 @@ -41,7 +41,7 @@ struct message_size pos; }; -static struct partial_cache partial = { 0, 0, 0, 0, { 0, 0, 0 } }; +static struct partial_cache last_partial = { 0, 0, 0, 0, { 0, 0, 0 } }; static bool seek_partial(unsigned int select_counter, unsigned int uid, struct partial_cache *partial, struct istream *stream, @@ -224,10 +224,10 @@ ctx->cur_offset += ret; if (ctx->update_partial) { - partial.cr_skipped = ctx->skip_cr != 0; - partial.pos.physical_size = - ctx->cur_input->v_offset - partial.physical_start; - partial.pos.virtual_size += ret; + last_partial.cr_skipped = ctx->skip_cr != 0; + last_partial.pos.physical_size = + ctx->cur_input->v_offset - last_partial.physical_start; + last_partial.pos.virtual_size += ret; } return ctx->cur_offset == ctx->cur_size; @@ -312,7 +312,7 @@ } else { ctx->skip_cr = seek_partial(ctx->select_counter, ctx->cur_mail->uid, - &partial, ctx->cur_input, body->skip); + &last_partial, ctx->cur_input, body->skip); } return fetch_stream(ctx, size);
--- a/src/imap/imap-search.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/imap-search.c Mon Jun 09 05:11:18 2008 +0300 @@ -42,6 +42,11 @@ struct mail_search_args *sargs; const char *error; + if (args->type == IMAP_ARG_EOL) { + client_send_command_error(cmd, "Missing search parameters"); + return -1; + } + if (mail_search_build_from_imap_args(args, charset, &sargs, &error) < 0) { client_send_command_error(cmd, error);
--- a/src/imap/imap-sync.c Sat May 17 17:50:54 2008 +0300 +++ b/src/imap/imap-sync.c Mon Jun 09 05:11:18 2008 +0300 @@ -168,6 +168,7 @@ STATUS_MESSAGES | STATUS_RECENT, &status) < 0 || ctx->failed) { mailbox_transaction_rollback(&ctx->t); + array_free(&ctx->tmp_keywords); i_free(ctx); return -1; } @@ -412,7 +413,7 @@ static bool cmd_sync_continue(struct client_command_context *sync_cmd) { - struct client_command_context *cmd; + struct client_command_context *cmd, *prev; struct client *client = sync_cmd->client; struct imap_sync_context *ctx = sync_cmd->context; int ret; @@ -431,13 +432,19 @@ } sync_cmd->context = NULL; - /* finish all commands that waited for this sync */ - for (cmd = client->command_queue; cmd != NULL; cmd = cmd->next) { + /* Finish all commands that waited for this sync. Go through the queue + backwards, so that tagged replies are sent in the same order as + they were received. This fixes problems with clients that rely on + this (Apple Mail 3.2) */ + for (cmd = client->command_queue; cmd->next != NULL; cmd = cmd->next) ; + for (; cmd != NULL; cmd = prev) { + prev = cmd->prev; + if (cmd->state == CLIENT_COMMAND_STATE_WAIT_SYNC && cmd != sync_cmd && cmd->sync->counter+1 == client->sync_counter) { if (cmd_finish_sync(cmd)) - client_command_free(cmd); + client_command_free(&cmd); } } return cmd_finish_sync(sync_cmd); @@ -465,10 +472,9 @@ count++; } } + i_assert(noexpunges_count == 0 || noexpunges_count == count); if (fast_count != count) *flags_r &= ~MAILBOX_SYNC_FLAG_FAST; - if (noexpunges_count != count) - *flags_r &= ~MAILBOX_SYNC_FLAG_NO_EXPUNGES; i_assert((*flags_r & (MAILBOX_SYNC_AUTO_STOP | MAILBOX_SYNC_FLAG_FIX_INCONSISTENT)) == 0); @@ -509,7 +515,7 @@ return FALSE; } - client_command_free(sync_cmd); + client_command_free(&sync_cmd); (void)cmd_sync_delayed(client); return TRUE; } @@ -566,16 +572,20 @@ static bool cmd_sync_drop_fast(struct client *client) { - struct client_command_context *cmd, *next; + struct client_command_context *cmd, *prev; bool ret = FALSE; - for (cmd = client->command_queue; cmd != NULL; cmd = next) { - next = cmd->next; + if (client->command_queue == NULL) + return FALSE; + + for (cmd = client->command_queue; cmd->next != NULL; cmd = cmd->next) ; + for (; cmd != NULL; cmd = prev) { + prev = cmd->next; if (cmd->state == CLIENT_COMMAND_STATE_WAIT_SYNC && (cmd->sync->flags & MAILBOX_SYNC_FLAG_FAST) != 0) { if (cmd_finish_sync(cmd)) { - client_command_free(cmd); + client_command_free(&cmd); ret = TRUE; } } @@ -585,7 +595,7 @@ bool cmd_sync_delayed(struct client *client) { - struct client_command_context *cmd; + struct client_command_context *cmd, *first_expunge, *first_nonexpunge; if (client->output_lock != NULL) { /* wait until we can send output to client */ @@ -599,13 +609,32 @@ return cmd_sync_drop_fast(client); } - /* find a command that we can sync */ + /* separate syncs that can send expunges from those that can't */ + first_expunge = first_nonexpunge = NULL; for (cmd = client->command_queue; cmd != NULL; cmd = cmd->next) { - if (cmd->state == CLIENT_COMMAND_STATE_WAIT_SYNC) { - if (cmd->sync->counter == client->sync_counter) - break; + if (cmd->sync != NULL && + cmd->sync->counter == client->sync_counter) { + if (cmd->sync->flags & MAILBOX_SYNC_FLAG_NO_EXPUNGES) { + if (first_nonexpunge == NULL) + first_nonexpunge = cmd; + } else { + if (first_expunge == NULL) + first_expunge = cmd; + } } } + if (first_expunge != NULL && first_nonexpunge != NULL) { + /* sync expunges after nonexpunges */ + for (cmd = first_expunge; cmd != NULL; cmd = cmd->next) { + if (cmd->sync != NULL && + cmd->sync->counter == client->sync_counter && + (cmd->sync->flags & + MAILBOX_SYNC_FLAG_NO_EXPUNGES) == 0) + cmd->sync->counter++; + } + first_expunge = NULL; + } + cmd = first_nonexpunge != NULL ? first_nonexpunge : first_expunge; if (cmd == NULL) return cmd_sync_drop_fast(client);
--- a/src/lib-dict/dict-db.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-dict/dict-db.c Mon Jun 09 05:11:18 2008 +0300 @@ -406,8 +406,9 @@ dict->pdb->del(dict->pdb, ctx->tid, &dkey, 0); } -static void db_dict_atomic_inc(struct dict_transaction_context *_ctx, - const char *key, long long diff) +static void +db_dict_atomic_inc(struct dict_transaction_context *_ctx ATTR_UNUSED, + const char *key ATTR_UNUSED, long long diff ATTR_UNUSED) { /* FIXME */ }
--- a/src/lib-dict/dict.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-dict/dict.c Mon Jun 09 05:11:18 2008 +0300 @@ -57,6 +57,8 @@ struct dict *dict; const char *p, *name; + i_assert(username != NULL); + p = strchr(uri, ':'); if (p == NULL) { i_error("Dictionary URI is missing ':': %s", uri);
--- a/src/lib-imap/imap-date.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-imap/imap-date.c Mon Jun 09 05:11:18 2008 +0300 @@ -105,7 +105,7 @@ return FALSE; } -bool imap_parse_date(const char *str, time_t *time) +bool imap_parse_date(const char *str, time_t *timestamp_r) { struct tm tm; @@ -114,11 +114,12 @@ return FALSE; tm.tm_isdst = -1; - (void)imap_mktime(&tm, time); + (void)imap_mktime(&tm, timestamp_r); return TRUE; } -bool imap_parse_datetime(const char *str, time_t *time, int *timezone_offset) +bool imap_parse_datetime(const char *str, time_t *timestamp_r, + int *timezone_offset_r) { struct tm tm; @@ -149,22 +150,22 @@ str += 3; /* timezone */ - *timezone_offset = parse_timezone(str); + *timezone_offset_r = parse_timezone(str); tm.tm_isdst = -1; - if (imap_mktime(&tm, time)) - *time -= *timezone_offset * 60; + if (imap_mktime(&tm, timestamp_r)) + *timestamp_r -= *timezone_offset_r * 60; return TRUE; } -const char *imap_to_datetime(time_t time) +const char *imap_to_datetime(time_t timestamp) { char *buf; struct tm *tm; int timezone_offset, year; - tm = localtime(&time); - timezone_offset = utc_offset(tm, time); + tm = localtime(×tamp); + timezone_offset = utc_offset(tm, timestamp); /* @UNSAFE: but faster than t_strdup_printf() call.. */ buf = t_malloc(27);
--- a/src/lib-imap/imap-date.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-imap/imap-date.h Mon Jun 09 05:11:18 2008 +0300 @@ -8,10 +8,11 @@ If date is too low or too high to fit to time_t, it's set to lowest/highest allowed value. This allows you to use the value directly for comparing timestamps. */ -bool imap_parse_date(const char *str, time_t *time); -bool imap_parse_datetime(const char *str, time_t *time, int *timezone_offset); +bool imap_parse_date(const char *str, time_t *timestamp_r); +bool imap_parse_datetime(const char *str, time_t *timestamp_r, + int *timezone_offset_r); /* Returns given UTC time in local timezone. */ -const char *imap_to_datetime(time_t time); +const char *imap_to_datetime(time_t timestamp); #endif
--- a/src/lib-index/mail-cache-compress.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-cache-compress.c Mon Jun 09 05:11:18 2008 +0300 @@ -199,28 +199,28 @@ used_fields_count = i; } else { for (i = used_fields_count = 0; i < orig_fields_count; i++) { - struct mail_cache_field_private *field = + struct mail_cache_field_private *priv = &cache->fields[i]; enum mail_cache_decision_type dec = - field->field.decision; + priv->field.decision; /* if the decision isn't forced and this field hasn't been accessed for a while, drop it */ if ((dec & MAIL_CACHE_DECISION_FORCED) == 0 && - (time_t)field->last_used < max_drop_time && - !field->adding) { + (time_t)priv->last_used < max_drop_time && + !priv->adding) { dec = MAIL_CACHE_DECISION_NO; - field->field.decision = dec; + priv->field.decision = dec; } /* drop all fields we don't want */ if ((dec & ~MAIL_CACHE_DECISION_FORCED) == - MAIL_CACHE_DECISION_NO && !field->adding) { - field->used = FALSE; - field->last_used = 0; + MAIL_CACHE_DECISION_NO && !priv->adding) { + priv->used = FALSE; + priv->last_used = 0; } - ctx.field_file_map[i] = !field->used ? + ctx.field_file_map[i] = !priv->used ? (uint32_t)-1 : used_fields_count++; } } @@ -414,7 +414,7 @@ /* once we're sure that the compression was successful, update the offsets */ - mail_index_ext_reset(trans, cache->ext_id, file_seq); + mail_index_ext_reset(trans, cache->ext_id, file_seq, TRUE); offsets = array_get(&ext_offsets, &count); for (i = 0; i < count; i++) { if (offsets[i] != 0) {
--- a/src/lib-index/mail-index-map.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-index-map.c Mon Jun 09 05:11:18 2008 +0300 @@ -26,7 +26,7 @@ size = EXT_GLOBAL_ALLOC_SIZE + initial_count * EXT_PER_ALLOC_SIZE; map->extension_pool = - pool_alloconly_create("map extensions", + pool_alloconly_create(MEMPOOL_GROWING"map extensions", nearest_power(size)); } else { p_clear(map->extension_pool); @@ -332,11 +332,11 @@ for (i = 0; i < array_count(&map->keyword_idx_map); i++) { const char *keyword = name + kw_rec[i].name_offset; const unsigned int *old_idx; - unsigned int idx; + unsigned int kw_idx; old_idx = array_idx(&map->keyword_idx_map, i); - if (!mail_index_keyword_lookup(index, keyword, &idx) || - idx != *old_idx) { + if (!mail_index_keyword_lookup(index, keyword, &kw_idx) || + kw_idx != *old_idx) { mail_index_set_error(index, "Corrupted index file %s: " "Keywords changed unexpectedly", index->filepath); @@ -348,7 +348,7 @@ i = array_count(&map->keyword_idx_map); for (; i < kw_hdr->keywords_count; i++) { const char *keyword = name + kw_rec[i].name_offset; - unsigned int idx; + unsigned int kw_idx; if (*keyword == '\0') { mail_index_set_error(index, "Corrupted index file %s: " @@ -356,8 +356,8 @@ index->filepath); return -1; } - mail_index_keyword_lookup_or_create(index, keyword, &idx); - array_append(&map->keyword_idx_map, &idx, 1); + mail_index_keyword_lookup_or_create(index, keyword, &kw_idx); + array_append(&map->keyword_idx_map, &kw_idx, 1); } return 0; } @@ -419,6 +419,17 @@ return TRUE; } +static void mail_index_map_clear_recent_flags(struct mail_index_map *map) +{ + struct mail_index_record *rec; + unsigned int i; + + for (i = 0; i < map->hdr.messages_count; i++) { + rec = MAIL_INDEX_MAP_IDX(map, i); + rec->flags &= ~MAIL_RECENT; + } +} + int mail_index_map_check_header(struct mail_index_map *map) { struct mail_index *index = map->index; @@ -446,13 +457,18 @@ if (hdr->seen_messages_count > hdr->messages_count || hdr->deleted_messages_count > hdr->messages_count) return 0; - if (hdr->minor_version == 0) { + switch (hdr->minor_version) { + case 0: /* upgrade silently from v1.0 */ - map->hdr.minor_version = MAIL_INDEX_MINOR_VERSION; map->hdr.unused_old_recent_messages_count = 0; if (hdr->first_recent_uid == 0) map->hdr.first_recent_uid = 1; index->need_recreate = TRUE; + /* fall through */ + case 1: + /* pre-v1.1.rc6: make sure the \Recent flags are gone */ + mail_index_map_clear_recent_flags(map); + map->hdr.minor_version = MAIL_INDEX_MINOR_VERSION; } if (hdr->first_recent_uid == 0 || hdr->first_recent_uid > hdr->next_uid || @@ -793,13 +809,15 @@ return mail_index_map_clone(&tmp_map); } +/* returns -1 = error, 0 = index files are unusable, + 1 = index files are usable or at least repairable */ static int mail_index_map_latest_file(struct mail_index *index) { struct mail_index_map *old_map, *new_map; struct stat st; unsigned int lock_id; uoff_t file_size; - bool use_mmap; + bool use_mmap, unusable = FALSE; int ret, try; ret = mail_index_reopen_if_changed(index); @@ -809,7 +827,7 @@ /* the index file is lost/broken. let's hope that we can build it from the transaction log. */ - return 0; + return 1; } /* the index file is still open, lock it */ @@ -843,6 +861,10 @@ ret = mail_index_read_map(new_map, file_size, &lock_id); mail_index_unlock(index, &lock_id); } + if (ret == 0) { + /* the index files are unusable */ + unusable = TRUE; + } for (try = 0; ret > 0; try++) { /* make sure the header is ok before using this mapping */ @@ -853,8 +875,13 @@ else if (mail_index_map_parse_keywords(new_map) < 0) ret = 0; } T_END; - if (ret != 0 || try == 2) + if (ret != 0 || try == 2) { + if (ret < 0) { + unusable = TRUE; + ret = 0; + } break; + } /* fsck and try again */ old_map = index->map; @@ -870,7 +897,7 @@ } if (ret <= 0) { mail_index_unmap(&new_map); - return ret; + return ret < 0 ? -1 : (unusable ? 0 : 1); } i_assert(new_map->rec_map->records != NULL); @@ -909,18 +936,24 @@ } if (ret == 0) { - /* try to open and read the latest index. if it fails for - any reason, we'll fallback to updating the existing mapping - from transaction logs (which we'll also do even if the - reopening succeeds) */ - (void)mail_index_map_latest_file(index); - - /* if we're creating the index file, we don't have any - logs yet */ - if (index->log->head != NULL && index->indexid != 0) { - /* and update the map with the latest changes from - transaction log */ - ret = mail_index_sync_map(&index->map, type, TRUE); + /* try to open and read the latest index. if it fails, we'll + fallback to updating the existing mapping from transaction + logs (which we'll also do even if the reopening succeeds). + if index files are unusable (e.g. major version change) + don't even try to use the transaction log. */ + if (mail_index_map_latest_file(index) == 0) { + /* make sure we don't try to open the file again */ + if (unlink(index->filepath) < 0 && errno != ENOENT) + mail_index_set_syscall_error(index, "unlink()"); + } else { + /* if we're creating the index file, we don't have any + logs yet */ + if (index->log->head != NULL && index->indexid != 0) { + /* and update the map with the latest changes + from transaction log */ + ret = mail_index_sync_map(&index->map, type, + TRUE); + } } }
--- a/src/lib-index/mail-index-private.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-index-private.h Mon Jun 09 05:11:18 2008 +0300 @@ -221,6 +221,7 @@ unsigned int syncing:1; unsigned int need_recreate:1; unsigned int modseqs_enabled:1; + unsigned int initial_create:1; }; extern struct mail_index_module_register mail_index_module_register;
--- a/src/lib-index/mail-index-sync-ext.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-index-sync-ext.c Mon Jun 09 05:11:18 2008 +0300 @@ -513,7 +513,8 @@ ctx->cur_ext_ignore = FALSE; } else { /* extension was reset and this transaction hadn't - yet seen it. ignore this update. */ + yet seen it. ignore this update (except for + resets). */ ctx->cur_ext_ignore = TRUE; } @@ -529,31 +530,13 @@ return 1; } -int mail_index_sync_ext_reset(struct mail_index_sync_map_ctx *ctx, - const struct mail_transaction_ext_reset *u) +static void mail_index_sync_ext_clear(struct mail_index_view *view, + struct mail_index_map *map, + struct mail_index_ext *ext) { - struct mail_index_view *view = ctx->view; - struct mail_index_map *map = view->map; - struct mail_index_ext_header *ext_hdr; - struct mail_index_ext *ext; struct mail_index_record *rec; uint32_t i; - if (ctx->cur_ext_map_idx == (uint32_t)-1) { - mail_index_sync_set_corrupted(ctx, - "Extension reset without intro prefix"); - return -1; - } - if (ctx->cur_ext_ignore) - return 1; - - /* a new index file will be created, so the old data won't be - accidentally used by other processes. */ - map = mail_index_sync_get_atomic_map(ctx); - - ext = array_idx_modifiable(&map->extensions, ctx->cur_ext_map_idx); - ext->reset_id = u->new_reset_id; - memset(buffer_get_space_unsafe(map->hdr_copy_buf, ext->hdr_offset, ext->hdr_size), 0, ext->hdr_size); map->hdr_base = map->hdr_copy_buf->data; @@ -565,10 +548,34 @@ } map->rec_map->write_seq_first = 1; map->rec_map->write_seq_last = view->map->rec_map->records_count; +} + +int mail_index_sync_ext_reset(struct mail_index_sync_map_ctx *ctx, + const struct mail_transaction_ext_reset *u) +{ + struct mail_index_map *map = ctx->view->map; + struct mail_index_ext_header *ext_hdr; + struct mail_index_ext *ext; + + if (ctx->cur_ext_map_idx == (uint32_t)-1) { + mail_index_sync_set_corrupted(ctx, + "Extension reset without intro prefix"); + return -1; + } + /* since we're resetting the extension, don't check cur_ext_ignore */ + + /* a new index file will be created, so the old data won't be + accidentally used by other processes. */ + map = mail_index_sync_get_atomic_map(ctx); + + ext = array_idx_modifiable(&map->extensions, ctx->cur_ext_map_idx); + ext->reset_id = u->new_reset_id; + + if (!u->preserve_data) + mail_index_sync_ext_clear(ctx->view, map, ext); ext_hdr = get_ext_header(map, ext); ext_hdr->reset_id = u->new_reset_id; - return 1; }
--- a/src/lib-index/mail-index-sync-update.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-index-sync-update.c Mon Jun 09 05:11:18 2008 +0300 @@ -539,15 +539,17 @@ break; } case MAIL_TRANSACTION_EXT_RESET: { - const struct mail_transaction_ext_reset *rec = data; + struct mail_transaction_ext_reset rec; - if (hdr->size != sizeof(*rec)) { + /* old versions have only new_reset_id */ + if (hdr->size < sizeof(uint32_t)) { mail_index_sync_set_corrupted(ctx, "ext reset: invalid record size"); ret = -1; break; } - ret = mail_index_sync_ext_reset(ctx, rec); + memcpy(&rec, data, I_MIN(hdr->size, sizeof(rec))); + ret = mail_index_sync_ext_reset(ctx, &rec); break; } case MAIL_TRANSACTION_EXT_HDR_UPDATE: { @@ -791,6 +793,9 @@ map->hdr_base = map->hdr_copy_buf->data; } + mail_transaction_log_view_get_prev_pos(view->log_view, + &prev_seq, &prev_offset); + mail_index_sync_map_init(&sync_map_ctx, view, type); if (reset) { /* Reset the entire index. Leave only indexid and
--- a/src/lib-index/mail-index-transaction-private.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-index-transaction-private.h Mon Jun 09 05:11:18 2008 +0300 @@ -51,8 +51,9 @@ struct mail_index_transaction_ext_hdr_update *); ARRAY_DEFINE(ext_rec_updates, ARRAY_TYPE(seq_array)); ARRAY_DEFINE(ext_resizes, struct mail_transaction_ext_intro); - ARRAY_DEFINE(ext_resets, uint32_t); + ARRAY_DEFINE(ext_resets, struct mail_transaction_ext_reset); ARRAY_DEFINE(ext_reset_ids, uint32_t); + ARRAY_DEFINE(ext_reset_atomic, uint32_t); ARRAY_DEFINE(keyword_updates, struct mail_index_transaction_keyword_update);
--- a/src/lib-index/mail-index-transaction.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-index-transaction.c Mon Jun 09 05:11:18 2008 +0300 @@ -75,6 +75,8 @@ array_free(&t->ext_resets); if (array_is_created(&t->ext_reset_ids)) array_free(&t->ext_reset_ids); + if (array_is_created(&t->ext_reset_atomic)) + array_free(&t->ext_reset_atomic); t->first_new_seq = mail_index_view_get_messages_count(t->view)+1; t->last_new_seq = 0; @@ -1087,18 +1089,36 @@ } void mail_index_ext_reset(struct mail_index_transaction *t, uint32_t ext_id, - uint32_t reset_id) + uint32_t reset_id, bool clear_data) { + struct mail_transaction_ext_reset reset; + i_assert(reset_id != 0); + memset(&reset, 0, sizeof(reset)); + reset.new_reset_id = reset_id; + reset.preserve_data = !clear_data; + mail_index_ext_set_reset_id(t, ext_id, reset_id); if (!array_is_created(&t->ext_resets)) i_array_init(&t->ext_resets, ext_id + 2); - array_idx_set(&t->ext_resets, ext_id, &reset_id); + array_idx_set(&t->ext_resets, ext_id, &reset); t->log_ext_updates = TRUE; } +void mail_index_ext_reset_inc(struct mail_index_transaction *t, uint32_t ext_id, + uint32_t prev_reset_id, bool clear_data) +{ + uint32_t expected_reset_id = prev_reset_id + 1; + + mail_index_ext_reset(t, ext_id, (uint32_t)-1, clear_data); + + if (!array_is_created(&t->ext_reset_atomic)) + i_array_init(&t->ext_reset_atomic, ext_id + 2); + array_idx_set(&t->ext_reset_atomic, ext_id, &expected_reset_id); +} + static bool mail_index_transaction_has_ext_changes(struct mail_index_transaction *t) { @@ -1123,11 +1143,11 @@ } } if (array_is_created(&t->ext_resets)) { - const uint32_t *ids; + const struct mail_transaction_ext_reset *resets; - ids = array_get(&t->ext_resets, &count); + resets = array_get(&t->ext_resets, &count); for (i = 0; i < count; i++) { - if (ids[i] != 0) + if (resets[i].new_reset_id != 0) return TRUE; } }
--- a/src/lib-index/mail-index.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-index.c Mon Jun 09 05:11:18 2008 +0300 @@ -343,19 +343,21 @@ /* Create a new indexid for us. If we're opening index into memory, index->map doesn't exist yet. */ index->indexid = ioloop_time; + index->initial_create = TRUE; if (index->map != NULL) index->map->hdr.indexid = index->indexid; } - ret = mail_transaction_log_create(index->log); + ret = mail_transaction_log_create(index->log, FALSE); + index->initial_create = FALSE; created = TRUE; } if (ret >= 0) { - ret = index->map != NULL ? 0 : mail_index_try_open(index); + ret = index->map != NULL ? 1 : mail_index_try_open(index); if (ret == 0) { - /* doesn't exist / corrupted */ + /* corrupted */ mail_transaction_log_close(index->log); - ret = mail_transaction_log_create(index->log); + ret = mail_transaction_log_create(index->log, TRUE); if (ret == 0) { if (index->map != NULL) mail_index_unmap(&index->map);
--- a/src/lib-index/mail-index.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-index.h Mon Jun 09 05:11:18 2008 +0300 @@ -6,7 +6,7 @@ #include "seq-range-array.h" #define MAIL_INDEX_MAJOR_VERSION 7 -#define MAIL_INDEX_MINOR_VERSION 1 +#define MAIL_INDEX_MINOR_VERSION 2 #define MAIL_INDEX_HEADER_MIN_SIZE 120 @@ -433,14 +433,21 @@ uint32_t hdr_size, uint16_t record_size, uint16_t record_align); -/* Reset extension records and header. Any updates for this extension which - were issued before the writer had seen this reset are discarded. reset_id is - used to figure this out, so it must be different every time. */ +/* Reset extension. Any updates for this extension which were issued before the + writer had seen this reset are discarded. reset_id is used to figure this + out, so it must be different every time. If clear_data=TRUE, records and + header is zeroed. */ void mail_index_ext_reset(struct mail_index_transaction *t, uint32_t ext_id, - uint32_t reset_id); -/* Discard existing extension updates and write new updates using the given - reset_id. The difference to mail_index_ext_reset() is that this doesn't - clear any existing record or header data. */ + uint32_t reset_id, bool clear_data); +/* Like mail_index_ext_reset(), but increase extension's reset_id atomically + when the transaction is being committed. If prev_reset_id doesn't match the + latest reset_id, the reset_id isn't increased and all extension changes are + ignored. */ +void mail_index_ext_reset_inc(struct mail_index_transaction *t, uint32_t ext_id, + uint32_t prev_reset_id, bool clear_data); +/* Discard existing extension updates in this transaction and write new updates + using the given reset_id. The difference to mail_index_ext_reset() is that + this doesn't clear any existing record or header data. */ void mail_index_ext_set_reset_id(struct mail_index_transaction *t, uint32_t ext_id, uint32_t reset_id); /* Get the current reset_id for given extension. Returns TRUE if it exists. */
--- a/src/lib-index/mail-transaction-log-append.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-transaction-log-append.c Mon Jun 09 05:11:18 2008 +0300 @@ -166,6 +166,58 @@ return buf; } +static void +ext_reset_update_atomic(struct mail_index_transaction *t, + uint32_t ext_id, uint32_t expected_reset_id) +{ + const struct mail_index_ext *map_ext; + struct mail_transaction_ext_reset *reset; + uint32_t idx, reset_id; + + if (!mail_index_map_get_ext_idx(t->view->index->map, ext_id, &idx)) { + /* new extension */ + reset_id = 1; + } else { + map_ext = array_idx(&t->view->index->map->extensions, idx); + reset_id = map_ext->reset_id + 1; + } + if (reset_id != expected_reset_id) { + /* ignore this extension update */ + mail_index_ext_set_reset_id(t, ext_id, 0); + return; + } + + if (reset_id == 0) + reset_id++; + + array_idx_set(&t->ext_reset_ids, ext_id, &reset_id); + + /* reseting existing data is optional */ + if (array_is_created(&t->ext_resets)) { + reset = array_idx_modifiable(&t->ext_resets, ext_id); + if (reset->new_reset_id == (uint32_t)-1) + reset->new_reset_id = reset_id; + } +} + +static void +transaction_update_atomic_reset_ids(struct mail_index_transaction *t) +{ + const uint32_t *expected_reset_ids; + unsigned int ext_id, count; + + if (!array_is_created(&t->ext_reset_atomic)) + return; + + expected_reset_ids = array_get(&t->ext_reset_atomic, &count); + for (ext_id = 0; ext_id < count; ext_id++) { + if (expected_reset_ids[ext_id] != 0) { + ext_reset_update_atomic(t, ext_id, + expected_reset_ids[ext_id]); + } + } +} + static void log_append_ext_intro(struct log_append_context *ctx, uint32_t ext_id, uint32_t reset_id) { @@ -257,7 +309,8 @@ unsigned int update_count, resize_count, ext_count = 0; unsigned int hdrs_count, reset_id_count, reset_count; uint32_t ext_id, reset_id; - const uint32_t *reset_ids, *reset; + const struct mail_transaction_ext_reset *reset; + const uint32_t *reset_ids; const ARRAY_TYPE(seq_array) *update; buffer_t *buf; @@ -304,15 +357,15 @@ } memset(&ext_reset, 0, sizeof(ext_reset)); - buf = buffer_create_data(pool_datastack_create(), &ext_reset, sizeof(ext_reset)); buffer_set_used_size(buf, sizeof(ext_reset)); for (ext_id = 0; ext_id < ext_count; ext_id++) { - ext_reset.new_reset_id = - ext_id < reset_count && reset[ext_id] != 0 ? - reset[ext_id] : 0; + if (ext_id < reset_count) + ext_reset = reset[ext_id]; + else + ext_reset.new_reset_id = 0; if ((ext_id < resize_count && resize[ext_id].name_size) || (ext_id < update_count && array_is_created(&update[ext_id])) || @@ -499,6 +552,18 @@ return -1; } + if (array_is_created(&t->ext_reset_atomic)) { + if (mail_index_map(t->view->index, + MAIL_INDEX_SYNC_HANDLER_HEAD) <= 0) + return -1; + transaction_update_atomic_reset_ids(t); + + if (!TRANSACTION_HAS_CHANGES(t)) { + /* we aborted the ext changes, nothing else to do */ + return 0; + } + } + file = log->head; if (file->sync_offset < file->buffer_offset)
--- a/src/lib-index/mail-transaction-log-file.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-transaction-log-file.c Mon Jun 09 05:11:18 2008 +0300 @@ -357,7 +357,8 @@ return 0; } if (file->hdr.indexid != file->log->index->indexid) { - if (file->log->index->indexid != 0) { + if (file->log->index->indexid != 0 && + !file->log->index->initial_create) { /* index file was probably just rebuilt and we don't know about it yet */ mail_transaction_log_file_set_corrupted(file, @@ -1065,8 +1066,9 @@ return 0; } - if ((uoff_t)st.st_size == file->mmap_size) { - /* we already have the whole file mmaped */ + if (file->buffer != NULL && file->buffer_offset <= start_offset && + (uoff_t)st.st_size == file->buffer_offset + file->buffer->used) { + /* we already have the whole file mapped */ if ((ret = mail_transaction_log_file_sync(file)) < 0) return 0; if (ret > 0)
--- a/src/lib-index/mail-transaction-log-view.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-transaction-log-view.c Mon Jun 09 05:11:18 2008 +0300 @@ -149,6 +149,17 @@ return -1; } + if (min_file_offset > 0 && + min_file_offset < view->log->files->hdr.hdr_size) { + /* log file offset is probably corrupted in the index file. */ + mail_transaction_log_view_set_corrupted(view, + "file_seq=%u, min_file_offset (%"PRIuUOFF_T + ") < hdr_size (%u)", + min_file_seq, min_file_offset, + view->tail->hdr.hdr_size); + return -1; + } + view->tail = view->head = file = NULL; for (seq = min_file_seq; seq <= max_file_seq; seq++) { if (file == NULL || file->hdr.file_seq != seq) {
--- a/src/lib-index/mail-transaction-log.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-transaction-log.c Mon Jun 09 05:11:18 2008 +0300 @@ -99,7 +99,7 @@ return 1; } -int mail_transaction_log_create(struct mail_transaction_log *log) +int mail_transaction_log_create(struct mail_transaction_log *log, bool reset) { struct mail_transaction_log_file *file; const char *path; @@ -125,7 +125,7 @@ mail_transaction_log_file_free(&log->open_file); } - if (mail_transaction_log_file_create(file, FALSE) < 0) { + if (mail_transaction_log_file_create(file, reset) < 0) { mail_transaction_log_file_free(&file); return -1; } @@ -187,7 +187,7 @@ log->head->hdr.indexid != log->index->indexid) { if (--log->head->refcount == 0) mail_transaction_log_file_free(&log->head); - (void)mail_transaction_log_create(log); + (void)mail_transaction_log_create(log, FALSE); } } @@ -291,7 +291,7 @@ someone deleted it manually while the index was open. try to handle this nicely by creating a new log file. */ file = log->head; - if (mail_transaction_log_create(log) < 0) + if (mail_transaction_log_create(log, FALSE) < 0) return -1; i_assert(file->refcount > 0); file->refcount--;
--- a/src/lib-index/mail-transaction-log.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-index/mail-transaction-log.h Mon Jun 09 05:11:18 2008 +0300 @@ -95,6 +95,8 @@ struct mail_transaction_ext_reset { uint32_t new_reset_id; + uint8_t preserve_data; + uint8_t unused_padding[3]; }; /* these are set for the last ext_intro */ @@ -120,7 +122,7 @@ is corrupted, -1 if there was some I/O error. */ int mail_transaction_log_open(struct mail_transaction_log *log); /* Create, or recreate, the transaction log. Returns 0 if ok, -1 if error. */ -int mail_transaction_log_create(struct mail_transaction_log *log); +int mail_transaction_log_create(struct mail_transaction_log *log, bool reset); /* Close all the open transactions log files. */ void mail_transaction_log_close(struct mail_transaction_log *log);
--- a/src/lib-mail/istream-header-filter.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-mail/istream-header-filter.c Mon Jun 09 05:11:18 2008 +0300 @@ -28,10 +28,12 @@ ARRAY_DEFINE(match_change_lines, unsigned int); unsigned int header_read:1; + unsigned int seen_eoh:1; unsigned int header_parsed:1; unsigned int exclude:1; unsigned int crlf:1; unsigned int hide_body:1; + unsigned int add_missing_eoh:1; }; header_filter_callback *null_header_filter_callback = NULL; @@ -115,6 +117,14 @@ cmp_uint) != NULL; } +static void add_eol(struct header_filter_istream *mstream) +{ + if (mstream->crlf) + buffer_append(mstream->hdr_buf, "\r\n", 2); + else + buffer_append_c(mstream->hdr_buf, '\n'); +} + static ssize_t read_header(struct header_filter_istream *mstream) { struct message_header_line *hdr; @@ -154,6 +164,7 @@ mstream->cur_line++; if (hdr->eoh) { + mstream->seen_eoh = TRUE; matched = TRUE; if (!mstream->header_parsed && mstream->callback != NULL) { @@ -164,10 +175,7 @@ if (!matched) continue; - if (mstream->crlf) - buffer_append(mstream->hdr_buf, "\r\n", 2); - else - buffer_append_c(mstream->hdr_buf, '\n'); + add_eol(mstream); continue; } @@ -207,13 +215,8 @@ } buffer_append(mstream->hdr_buf, hdr->value, hdr->value_len); - if (!hdr->no_newline) { - if (mstream->crlf) { - buffer_append(mstream->hdr_buf, - "\r\n", 2); - } else - buffer_append_c(mstream->hdr_buf, '\n'); - } + if (!hdr->no_newline) + add_eol(mstream); if (mstream->skip_count >= mstream->hdr_buf->used) { /* we need more */ @@ -230,6 +233,11 @@ } } + if (hdr_ret < 0 && !mstream->seen_eoh && mstream->add_missing_eoh) { + mstream->seen_eoh = TRUE; + add_eol(mstream); + } + /* don't copy eof here because we're only returning headers here. the body will be returned in separate read() call. */ mstream->istream.buffer = buffer_get_data(mstream->hdr_buf, &pos); @@ -349,6 +357,7 @@ mstream->skip_count = v_offset; mstream->cur_line = 0; mstream->header_read = FALSE; + mstream->seen_eoh = FALSE; } else { /* body */ v_offset += mstream->header_size.physical_size - @@ -398,7 +407,8 @@ i_assert((flags & (HEADER_FILTER_INCLUDE|HEADER_FILTER_EXCLUDE)) != 0); mstream = i_new(struct header_filter_istream, 1); - mstream->pool = pool_alloconly_create("header filter stream", 4096); + mstream->pool = pool_alloconly_create(MEMPOOL_GROWING + "header filter stream", 4096); mstream->istream.max_buffer_size = input->real_stream->max_buffer_size; mstream->headers = headers_count == 0 ? NULL : @@ -413,6 +423,7 @@ mstream->exclude = (flags & HEADER_FILTER_EXCLUDE) != 0; mstream->crlf = (flags & HEADER_FILTER_NO_CR) == 0; mstream->hide_body = (flags & HEADER_FILTER_HIDE_BODY) != 0; + mstream->add_missing_eoh = (flags & HEADER_FILTER_ADD_MISSING_EOH) != 0; mstream->istream.iostream.destroy = i_stream_header_filter_destroy; mstream->istream.iostream.set_max_buffer_size =
--- a/src/lib-mail/istream-header-filter.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-mail/istream-header-filter.h Mon Jun 09 05:11:18 2008 +0300 @@ -3,14 +3,16 @@ enum header_filter_flags { /* Include only specified headers in output.*/ - HEADER_FILTER_INCLUDE = 0x01, + HEADER_FILTER_INCLUDE = 0x01, /* Exclude specified headers from output. */ - HEADER_FILTER_EXCLUDE = 0x02, + HEADER_FILTER_EXCLUDE = 0x02, /* Use LF linefeeds instead of CRLF. */ - HEADER_FILTER_NO_CR = 0x04, + HEADER_FILTER_NO_CR = 0x04, /* Return EOF at the beginning of message body. */ - HEADER_FILTER_HIDE_BODY = 0x08 + HEADER_FILTER_HIDE_BODY = 0x08, + /* If the empty "end of headers" line doesn't exist, add it. */ + HEADER_FILTER_ADD_MISSING_EOH = 0x10 }; struct message_header_line;
--- a/src/lib-mail/mbox-from.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-mail/mbox-from.c Mon Jun 09 05:11:18 2008 +0300 @@ -54,7 +54,7 @@ { const unsigned char *msg_start, *sender_end, *msg_end; struct tm tm; - int esc, alt_stamp, timezone = 0, seen_timezone = FALSE; + int esc, alt_stamp, timezone_secs = 0, seen_timezone = FALSE; time_t t; *time_r = (time_t)-1; @@ -198,9 +198,9 @@ i_isdigit(msg[3]) && i_isdigit(msg[4]) && msg[5] == ' ') { /* numeric timezone, use it */ seen_timezone = TRUE; - timezone = (msg[1]-'0') * 10*60*60 + (msg[2]-'0') * 60*60 + + timezone_secs = (msg[1]-'0') * 10*60*60 + (msg[2]-'0') * 60*60 + (msg[3]-'0') * 10 + (msg[4]-'0'); - if (msg[0] == '-') timezone = -timezone; + if (msg[0] == '-') timezone_secs = -timezone_secs; msg += 6; } @@ -217,9 +217,9 @@ i_isdigit(msg[2]) && i_isdigit(msg[3]) && i_isdigit(msg[4]) && i_isdigit(msg[5])) { seen_timezone = TRUE; - timezone = (msg[2]-'0') * 10*60*60 + (msg[3]-'0') * 60*60 + + timezone_secs = (msg[2]-'0') * 10*60*60 + (msg[3]-'0') * 60*60 + (msg[4]-'0') * 10 + (msg[5]-'0'); - if (msg[1] == '-') timezone = -timezone; + if (msg[1] == '-') timezone_secs = -timezone_secs; } if (seen_timezone) { @@ -227,7 +227,7 @@ if (t == (time_t)-1) return -1; - t -= timezone; + t -= timezone_secs; *time_r = t; } else { /* assume local timezone */ @@ -238,7 +238,7 @@ return 0; } -const char *mbox_from_create(const char *sender, time_t time) +const char *mbox_from_create(const char *sender, time_t timestamp) { string_t *str; struct tm *tm; @@ -251,7 +251,7 @@ /* we could use simply asctime(), but i18n etc. may break it. Example: "Thu Nov 29 22:33:52 2001" */ - tm = localtime(&time); + tm = localtime(×tamp); /* week day */ str_append(str, weekdays[tm->tm_wday]);
--- a/src/lib-mail/mbox-from.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-mail/mbox-from.h Mon Jun 09 05:11:18 2008 +0300 @@ -7,6 +7,6 @@ time_t *time_r, char **sender_r); /* Return a mbox-compatible From_-line using given sender and time. The returned string begins with "From ". */ -const char *mbox_from_create(const char *sender, time_t time); +const char *mbox_from_create(const char *sender, time_t timestamp); #endif
--- a/src/lib-mail/message-date.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-mail/message-date.c Mon Jun 09 05:11:18 2008 +0300 @@ -105,8 +105,9 @@ return ret < 0 ? -1 : *value_len > 0; } -static bool message_date_parser_tokens(struct message_date_parser_context *ctx, - time_t *time, int *timezone_offset) +static bool +message_date_parser_tokens(struct message_date_parser_context *ctx, + time_t *timestamp_r, int *timezone_offset_r) { struct tm tm; const unsigned char *value; @@ -212,24 +213,24 @@ return FALSE; if (ret == 0) { /* missing timezone */ - *timezone_offset = 0; + *timezone_offset_r = 0; } else { /* timezone */ - *timezone_offset = parse_timezone(value, len); + *timezone_offset_r = parse_timezone(value, len); } tm.tm_isdst = -1; - *time = utc_mktime(&tm); - if (*time == (time_t)-1) + *timestamp_r = utc_mktime(&tm); + if (*timestamp_r == (time_t)-1) return FALSE; - *time -= *timezone_offset * 60; + *timestamp_r -= *timezone_offset_r * 60; return TRUE; } bool message_date_parse(const unsigned char *data, size_t size, - time_t *time, int *timezone_offset) + time_t *timestamp_r, int *timezone_offset_r) { bool success; @@ -238,21 +239,21 @@ rfc822_parser_init(&ctx.parser, data, size, NULL); ctx.str = t_str_new(128); - success = message_date_parser_tokens(&ctx, time, - timezone_offset); + success = message_date_parser_tokens(&ctx, timestamp_r, + timezone_offset_r); } T_END; return success; } -const char *message_date_create(time_t time) +const char *message_date_create(time_t timestamp) { struct tm *tm; int offset; bool negative; - tm = localtime(&time); - offset = utc_offset(tm, time); + tm = localtime(×tamp); + offset = utc_offset(tm, timestamp); if (offset >= 0) negative = FALSE; else {
--- a/src/lib-mail/message-date.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-mail/message-date.h Mon Jun 09 05:11:18 2008 +0300 @@ -4,9 +4,9 @@ /* Parses RFC2822 date/time string. timezone_offset is filled with the timezone's difference to UTC in minutes. */ bool message_date_parse(const unsigned char *data, size_t size, - time_t *time, int *timezone_offset); + time_t *timestamp_r, int *timezone_offset_r); /* Create RFC2822 date/time string from given time in local timezone. */ -const char *message_date_create(time_t time); +const char *message_date_create(time_t timestamp); #endif
--- a/src/lib-mail/message-parser.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-mail/message-parser.c Mon Jun 09 05:11:18 2008 +0300 @@ -617,7 +617,11 @@ uoff_t offset = ctx->part->physical_pos + ctx->part->header_size.physical_size; - i_assert(offset >= ctx->input->v_offset); + if (offset < ctx->input->v_offset) { + /* header was actually larger than the cached size suggested */ + ctx->broken = TRUE; + return -1; + } i_stream_skip(ctx->input, offset - ctx->input->v_offset); ctx->parse_next_block = preparsed_parse_body_more;
--- a/src/lib-sql/driver-mysql.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-sql/driver-mysql.c Mon Jun 09 05:11:18 2008 +0300 @@ -534,10 +534,11 @@ } static const unsigned char * -driver_mysql_result_get_field_value_binary(struct sql_result *_result, - unsigned int idx, size_t *size_r) +driver_mysql_result_get_field_value_binary(struct sql_result *_result ATTR_UNUSED, + unsigned int idx ATTR_UNUSED, + size_t *size_r ATTR_UNUSED) { - // FIXME + /* FIXME */ return NULL; }
--- a/src/lib-storage/index/Makefile.am Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/Makefile.am Mon Jun 09 05:11:18 2008 +0300 @@ -16,6 +16,7 @@ index-mailbox-check.c \ index-search.c \ index-sort.c \ + index-sort-string.c \ index-status.c \ index-storage.c \ index-sync.c \ @@ -29,6 +30,7 @@ headers = \ index-mail.h \ index-sort.h \ + index-sort-private.h \ index-storage.h \ index-sync-changes.h \ index-sync-private.h \
--- a/src/lib-storage/index/cydir/cydir-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/cydir/cydir-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -28,6 +28,7 @@ cydir_list_delete_mailbox(struct mailbox_list *list, const char *name); static int cydir_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name, enum mailbox_list_file_type type, enum mailbox_info_flags *flags); @@ -335,6 +336,7 @@ static int cydir_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx ATTR_UNUSED, const char *dir, const char *fname, + const char *mailbox_name ATTR_UNUSED, enum mailbox_list_file_type type, enum mailbox_info_flags *flags) { @@ -374,6 +376,9 @@ if (st.st_nlink > 2) *flags |= MAILBOX_CHILDREN; } + } else if (errno == ENOENT) { + /* doesn't exist - probably a non-existing subscribed mailbox */ + *flags |= MAILBOX_NONEXISTENT; } else { /* non-selectable. probably either access denied, or symlink destination not found. don't bother logging errors. */
--- a/src/lib-storage/index/dbox/dbox-file-maildir.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/dbox/dbox-file-maildir.c Mon Jun 09 05:11:18 2008 +0300 @@ -40,11 +40,13 @@ { struct stat st; uoff_t size; + const char *value = NULL; switch (key) { case DBOX_METADATA_FLAGS: case DBOX_METADATA_KEYWORDS: - return dbox_file_maildir_get_flags(file, key); + value = dbox_file_maildir_get_flags(file, key); + break; case DBOX_METADATA_RECEIVED_TIME: case DBOX_METADATA_SAVE_TIME: if (file->fd != -1) { @@ -61,17 +63,23 @@ } } if (key == DBOX_METADATA_RECEIVED_TIME) - return dec2str(st.st_mtime); + value = dec2str(st.st_mtime); else - return dec2str(st.st_ctime); + value = dec2str(st.st_ctime); + break; case DBOX_METADATA_VIRTUAL_SIZE: - maildir_filename_get_size(file->fname, - MAILDIR_EXTRA_VIRTUAL_SIZE, &size); - return dec2str(size); + if (maildir_filename_get_size(file->fname, + MAILDIR_EXTRA_VIRTUAL_SIZE, + &size)) + value = dec2str(size); + break; + case DBOX_METADATA_POP3_UIDL: case DBOX_METADATA_EXPUNGED: case DBOX_METADATA_EXT_REF: case DBOX_METADATA_SPACE: break; } - return NULL; + if (value != NULL) + dbox_file_metadata_set(file, key, value); + return value; }
--- a/src/lib-storage/index/dbox/dbox-file.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/dbox/dbox-file.c Mon Jun 09 05:11:18 2008 +0300 @@ -600,7 +600,10 @@ return 0; } - *uid_r = hex2dec(hdr.uid_hex, sizeof(hdr.uid_hex)); + /* Ignore the UID header with UID files */ + *uid_r = (file->file_id & DBOX_FILE_ID_FLAG_UID) != 0 ? + (file->file_id & ~DBOX_FILE_ID_FLAG_UID) : + hex2dec(hdr.uid_hex, sizeof(hdr.uid_hex)); *physical_size_r = hex2dec(hdr.message_size_hex, sizeof(hdr.message_size_hex)); return 1; @@ -623,7 +626,7 @@ if (offset == 0) offset = file->file_header_size; - if (offset != file->cur_offset) { + if (offset != file->cur_offset || file->cur_uid == 0) { file->cur_offset = offset; i_stream_seek(file->input, offset); ret = dbox_file_read_mail_header(file, &file->cur_uid, @@ -757,8 +760,13 @@ { int ret; - if (file->nonappendable) - return 0; + if (file->append_count == 0) { + if (file->nonappendable) + return 0; + } else { + if (!dbox_file_can_append(file, mail_size)) + return 0; + } ret = dbox_file_get_append_stream_int(file, mail_size, stream_r); if (ret == 0) @@ -940,7 +948,7 @@ const char **changes, *data; unsigned int i, count; - data = dbox_file_metadata_get(file, key); + data = file->maildir_file ? NULL : dbox_file_metadata_get(file, key); if (data != NULL && strcmp(data, value) == 0) { /* value didn't change */ return;
--- a/src/lib-storage/index/dbox/dbox-file.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/dbox/dbox-file.h Mon Jun 09 05:11:18 2008 +0300 @@ -54,15 +54,17 @@ DBOX_METADATA_FLAGS = 'F', /* Space separated list of keywords */ DBOX_METADATA_KEYWORDS = 'K', - /* Pointer to external message data. Format is: - 1*(<start offset> <byte count> <ref>) */ - DBOX_METADATA_EXT_REF = 'P', + /* POP3 UIDL overriding the default format */ + DBOX_METADATA_POP3_UIDL = 'P', /* Received UNIX timestamp in hex */ DBOX_METADATA_RECEIVED_TIME = 'R', /* Saved UNIX timestamp in hex */ DBOX_METADATA_SAVE_TIME = 'S', /* Virtual message size in hex (line feeds counted as CRLF) */ DBOX_METADATA_VIRTUAL_SIZE = 'V', + /* Pointer to external message data. Format is: + 1*(<start offset> <byte count> <ref>) */ + DBOX_METADATA_EXT_REF = 'X', /* End of metadata block. The spaces can be used for writing more metadata. */
--- a/src/lib-storage/index/dbox/dbox-index.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/dbox/dbox-index.c Mon Jun 09 05:11:18 2008 +0300 @@ -155,11 +155,11 @@ struct dbox_index_file_header *hdr) { if (index->uid_validity == 0) { - const struct mail_index_header *hdr; + const struct mail_index_header *idx_hdr; - hdr = mail_index_get_header(index->mbox->ibox.view); - index->uid_validity = hdr->uid_validity != 0 ? - hdr->uid_validity : (uint32_t)ioloop_time; + idx_hdr = mail_index_get_header(index->mbox->ibox.view); + index->uid_validity = idx_hdr->uid_validity != 0 ? + idx_hdr->uid_validity : (uint32_t)ioloop_time; } memset(hdr, ' ', sizeof(*hdr)); @@ -346,6 +346,7 @@ off_t start, off_t len) { struct flock fl; + const char *errstr; fl.l_type = lock_type; fl.l_whence = SEEK_SET; @@ -355,9 +356,12 @@ if ((errno == EACCES || errno == EAGAIN || errno == EINTR) && cmd == F_SETLK) return 0; + + errstr = errno != EACCES ? strerror(errno) : + "File is locked by another process (EACCES)"; mail_storage_set_critical(index->mbox->ibox.box.storage, - "fcntl(%s, %s) failed: %m", index->path, - lock_type == F_UNLCK ? "F_UNLCK" : "F_WRLCK"); + "fcntl(%s, %s) failed: %s", index->path, + lock_type == F_UNLCK ? "F_UNLCK" : "F_WRLCK", errstr); return -1; } return 1;
--- a/src/lib-storage/index/dbox/dbox-mail.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/dbox/dbox-mail.c Mon Jun 09 05:11:18 2008 +0300 @@ -3,6 +3,7 @@ #include "lib.h" #include "ioloop.h" #include "istream.h" +#include "str.h" #include "index-mail.h" #include "dbox-storage.h" #include "dbox-file.h" @@ -177,6 +178,47 @@ } static int +dbox_mail_get_special(struct mail *_mail, enum mail_fetch_field field, + const char **value_r) +{ + struct dbox_mail *mail = (struct dbox_mail *)_mail; + struct index_mail *imail = &mail->imail; + const unsigned int pop3_uidl_cache_field = + imail->ibox->cache_fields[MAIL_CACHE_POP3_UIDL].idx; + struct dbox_file *file; + const char *value; + string_t *str; + + switch (field) { + case MAIL_FETCH_UIDL_BACKEND: + /* keep the UIDL in cache file, otherwise POP3 would open all + mail files and read the metadata */ + str = str_new(imail->data_pool, 64); + if (mail_cache_lookup_field(imail->trans->cache_view, str, + _mail->seq, + pop3_uidl_cache_field) > 0) { + *value_r = str_c(str); + return 0; + } + + if (dbox_mail_metadata_seek(mail, &file) < 0) + return -1; + + value = dbox_file_metadata_get(file, DBOX_METADATA_POP3_UIDL); + if (value == NULL) + value = ""; + index_mail_cache_add_idx(imail, pop3_uidl_cache_field, + value, strlen(value)+1); + *value_r = value; + return 0; + default: + break; + } + + return index_mail_get_special(_mail, field, value_r); +} + +static int dbox_mail_get_stream(struct mail *_mail, struct message_size *hdr_size, struct message_size *body_size, struct istream **stream_r) { @@ -204,6 +246,9 @@ /* FIXME: broken file/offset */ if (ret > 0) i_stream_unref(&input); + mail_storage_set_critical(_mail->box->storage, + "broken pointer to dbox file %s", + mail->open_file->current_path); return -1; } data->physical_size = size; @@ -234,7 +279,7 @@ index_mail_get_headers, index_mail_get_header_stream, dbox_mail_get_stream, - index_mail_get_special, + dbox_mail_get_special, index_mail_update_flags, index_mail_update_keywords, index_mail_expunge,
--- a/src/lib-storage/index/dbox/dbox-save.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/dbox/dbox-save.c Mon Jun 09 05:11:18 2008 +0300 @@ -2,6 +2,7 @@ #include "lib.h" #include "array.h" +#include "fdatasync-path.h" #include "hex-dec.h" #include "str.h" #include "istream.h" @@ -218,6 +219,29 @@ o_stream_send(ctx->cur_output, "\n", 1); } +static int dbox_save_mail_write_header(struct dbox_save_mail *mail) +{ + struct dbox_message_header dbox_msg_hdr; + + i_assert(mail->file->msg_header_size == sizeof(dbox_msg_hdr)); + + mail->file->last_append_uid = mail->uid; + dbox_msg_header_fill(&dbox_msg_hdr, mail->uid, mail->message_size); + + if (pwrite_full(mail->file->fd, &dbox_msg_hdr, + sizeof(dbox_msg_hdr), mail->append_offset) < 0) { + dbox_file_set_syscall_error(mail->file, "write"); + return -1; + } + if (!mail->file->mbox->ibox.fsync_disable) { + if (fdatasync(mail->file->fd) < 0) { + dbox_file_set_syscall_error(mail->file, "fdatasync"); + return -1; + } + } + return 0; +} + int dbox_save_finish(struct mail_save_context *_ctx) { struct dbox_save_context *ctx = (struct dbox_save_context *)_ctx; @@ -250,12 +274,6 @@ i_stream_unref(&ctx->input); count = array_count(&ctx->mails); - if (count >= ctx->mbox->max_open_files) { - /* too many open files, close one of them */ - save_mail = array_idx_modifiable(&ctx->mails, count - - ctx->mbox->max_open_files); - dbox_file_close(save_mail->file); - } save_mail = array_idx_modifiable(&ctx->mails, count - 1); if (ctx->failed) { dbox_file_cancel_append(save_mail->file, @@ -267,6 +285,12 @@ dbox_file_finish_append(save_mail->file); save_mail->message_size = offset - save_mail->append_offset - save_mail->file->msg_header_size; + + if (save_mail->file->append_count == 1 && + !dbox_file_can_append(save_mail->file, 0)) { + dbox_save_mail_write_header(save_mail); + dbox_file_close(save_mail->file); + } return 0; } } @@ -279,23 +303,6 @@ (void)dbox_save_finish(_ctx); } -static int dbox_save_mail_write_header(struct dbox_save_mail *mail) -{ - struct dbox_message_header dbox_msg_hdr; - - i_assert(mail->file->msg_header_size == sizeof(dbox_msg_hdr)); - - mail->file->last_append_uid = mail->uid; - dbox_msg_header_fill(&dbox_msg_hdr, mail->uid, mail->message_size); - - if (pwrite_full(mail->file->fd, &dbox_msg_hdr, - sizeof(dbox_msg_hdr), mail->append_offset) < 0) { - dbox_file_set_syscall_error(mail->file, "write"); - return -1; - } - return 0; -} - static int dbox_save_file_write_append_offset(struct dbox_file *file, uoff_t append_offset) { @@ -359,6 +366,14 @@ /* update headers */ qsort(mails, count, sizeof(*mails), dbox_save_mail_file_cmp); for (i = 0; i < count; i++) { + mails[i].file->last_append_uid = mails[i].uid; + if (mails[i].file->append_count == 1 && + !dbox_file_can_append(mails[i].file, 0)) { + /* UID file - there's no need to write it to the + header */ + continue; + } + if (dbox_file_open_if_needed(mails[i].file) < 0 || dbox_save_mail_write_header(&mails[i]) < 0) { ret = -1; @@ -444,6 +459,13 @@ ctx->ctx.transaction = NULL; /* transaction is already freed */ (void)dbox_sync_finish(&ctx->sync_ctx, TRUE); + + if (!ctx->mbox->ibox.fsync_disable) { + if (fdatasync_path(ctx->mbox->path) < 0) { + i_error("fdatasync_path(%s) failed: %m", + ctx->mbox->path); + } + } dbox_transaction_save_rollback(ctx); }
--- a/src/lib-storage/index/dbox/dbox-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/dbox/dbox-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -40,6 +40,7 @@ const char *oldname, const char *newname); static int dbox_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name, enum mailbox_list_file_type type, enum mailbox_info_flags *flags); @@ -306,9 +307,12 @@ static int dbox_storage_mailbox_close(struct mailbox *box) { struct dbox_mailbox *mbox = (struct dbox_mailbox *)box; - int ret; + int ret = 0; - ret = dbox_sync(mbox, TRUE); + if (box->opened) { + /* see if we want to flush dirty flags */ + ret = dbox_sync(mbox, TRUE); + } dbox_index_deinit(&mbox->dbox_index); dbox_files_free(mbox); @@ -318,13 +322,14 @@ } static int dbox_mailbox_create(struct mail_storage *_storage, - const char *name, bool directory ATTR_UNUSED) + const char *name, bool directory) { struct dbox_storage *storage = (struct dbox_storage *)_storage; const char *path, *alt_path; struct stat st; path = mailbox_list_get_path(_storage->list, name, + directory ? MAILBOX_LIST_PATH_TYPE_DIR : MAILBOX_LIST_PATH_TYPE_MAILBOX); if (stat(path, &st) == 0) { mail_storage_set_error(_storage, MAIL_ERROR_NOTPOSSIBLE, @@ -336,7 +341,7 @@ race conditions with RENAME/DELETE), but if something crashed and left it lying around we don't want to start overwriting files in it. */ - alt_path = dbox_get_alt_path(storage, path); + alt_path = directory ? NULL : dbox_get_alt_path(storage, path); if (alt_path != NULL && stat(alt_path, &st) == 0) { mail_storage_set_error(_storage, MAIL_ERROR_NOTPOSSIBLE, "Mailbox already exists"); @@ -578,6 +583,7 @@ static int dbox_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx ATTR_UNUSED, const char *dir, const char *fname, + const char *mailbox_name ATTR_UNUSED, enum mailbox_list_file_type type, enum mailbox_info_flags *flags) { @@ -617,12 +623,15 @@ if (st.st_nlink > 2) *flags |= MAILBOX_CHILDREN; } + } else if (errno == ENOENT) { + /* doesn't exist - probably a non-existing subscribed mailbox */ + *flags |= MAILBOX_NONEXISTENT; } else { /* non-selectable. probably either access denied, or symlink destination not found. don't bother logging errors. */ *flags |= MAILBOX_NOSELECT; } - if ((*flags & MAILBOX_NOSELECT) == 0) { + if ((*flags & (MAILBOX_NOSELECT | MAILBOX_NONEXISTENT)) == 0) { /* make sure it's a selectable mailbox */ maildir_path = t_strconcat(path, "/"DBOX_MAILDIR_NAME, NULL); if (stat(maildir_path, &st) < 0 || !S_ISDIR(st.st_mode))
--- a/src/lib-storage/index/dbox/dbox-sync-file.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/dbox/dbox-sync-file.c Mon Jun 09 05:11:18 2008 +0300 @@ -196,15 +196,22 @@ dbox_sync_file_split(struct dbox_sync_context *ctx, struct dbox_file *in_file, uoff_t offset, uint32_t seq) { + static enum dbox_metadata_key maildir_metadata_keys[] = { + DBOX_METADATA_VIRTUAL_SIZE, + DBOX_METADATA_RECEIVED_TIME, + DBOX_METADATA_SAVE_TIME, + DBOX_METADATA_POP3_UIDL + }; struct dbox_index_append_context *append_ctx; struct dbox_file *out_file; struct istream *input; struct ostream *output; struct dbox_message_header dbox_msg_hdr; struct dbox_mail_index_record rec; - const char *out_path; + const char *out_path, *value; uint32_t uid; uoff_t size, append_offset; + unsigned int i; int ret; bool expunged; @@ -233,6 +240,16 @@ dbox_sync_update_metadata(ctx, out_file, NULL, seq); } T_END; + /* set static metadata */ + for (i = 0; i < N_ELEMENTS(maildir_metadata_keys); i++) { + value = dbox_file_metadata_get(in_file, + maildir_metadata_keys[i]); + if (value != NULL) { + dbox_file_metadata_set(out_file, + maildir_metadata_keys[i], value); + } + } + /* copy the message */ out_path = dbox_file_get_path(out_file); o_stream_cork(output);
--- a/src/lib-storage/index/dbox/dbox-sync-rebuild.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/dbox/dbox-sync-rebuild.c Mon Jun 09 05:11:18 2008 +0300 @@ -70,7 +70,7 @@ ctx->cache_used = TRUE; ctx->cache_reset_id = reset_id; mail_index_ext_reset(ctx->trans, ctx->cache_ext_id, - ctx->cache_reset_id); + ctx->cache_reset_id, TRUE); } if (ctx->cache_reset_id == reset_id) { mail_index_update_ext(ctx->trans, new_seq, @@ -225,8 +225,9 @@ } static int -dbox_sync_index_multi_file(struct dbox_sync_rebuild_context *ctx, - const char *dir, const char *fname) +dbox_sync_index_multi_file(struct dbox_sync_rebuild_context *ctx ATTR_UNUSED, + const char *dir ATTR_UNUSED, + const char *fname ATTR_UNUSED) { /* FIXME */ return 0; @@ -253,8 +254,12 @@ } file = dbox_file_init_new_maildir(ctx->mbox, fname); - if ((ret = dbox_sync_index_file_next(ctx, file, &offset)) > 0) + if ((ret = dbox_sync_index_file_next(ctx, file, &offset)) > 0) { dbox_index_append_file(ctx->append_ctx, file); + /* appending referenced the file, so make sure it gets closed + so we don't have too many open files. */ + dbox_file_close(file); + } dbox_file_unref(&file); return ret < 0 ? -1 : 0; } @@ -413,6 +418,7 @@ MAIL_INDEX_TRANSACTION_FLAG_EXTERNAL); i_array_init(&ctx.maildir_new_files, 8); mail_index_reset(ctx.trans); + index_mailbox_reset_uidvalidity(&mbox->ibox); mail_index_ext_lookup(mbox->ibox.index, "cache", &ctx.cache_ext_id); if ((ret = dbox_sync_index_rebuild_ctx(&ctx)) < 0)
--- a/src/lib-storage/index/index-mail-headers.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/index-mail-headers.c Mon Jun 09 05:11:18 2008 +0300 @@ -235,7 +235,8 @@ } } - if ((mail->data.cache_fetch_fields & MAIL_FETCH_DATE) != 0) { + if ((mail->data.cache_fetch_fields & MAIL_FETCH_DATE) != 0 || + mail->data.save_sent_date) { array_idx_set(&mail->header_match, get_header_field_idx(mail->ibox, "Date"), &mail->header_match_value);
--- a/src/lib-storage/index/index-mail.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/index-mail.c Mon Jun 09 05:11:18 2008 +0300 @@ -32,6 +32,7 @@ { "imap.body", 0, MAIL_CACHE_FIELD_STRING, 0, 0 }, { "imap.bodystructure", 0, MAIL_CACHE_FIELD_STRING, 0, 0 }, { "imap.envelope", 0, MAIL_CACHE_FIELD_STRING, 0, 0 }, + { "pop3.uidl", 0, MAIL_CACHE_FIELD_STRING, 0, 0 }, { "mime.parts", 0, MAIL_CACHE_FIELD_VARIABLE_SIZE, 0, 0 } }; @@ -1005,6 +1006,7 @@ return 0; case MAIL_FETCH_FROM_ENVELOPE: case MAIL_FETCH_UIDL_FILE_NAME: + case MAIL_FETCH_UIDL_BACKEND: *value_r = ""; return 0; case MAIL_FETCH_HEADER_MD5:
--- a/src/lib-storage/index/index-mail.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/index-mail.h Mon Jun 09 05:11:18 2008 +0300 @@ -18,6 +18,7 @@ MAIL_CACHE_IMAP_BODY, MAIL_CACHE_IMAP_BODYSTRUCTURE, MAIL_CACHE_IMAP_ENVELOPE, + MAIL_CACHE_POP3_UIDL, MAIL_CACHE_MESSAGE_PARTS, MAIL_INDEX_CACHE_FIELD_COUNT
--- a/src/lib-storage/index/index-search.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/index-search.c Mon Jun 09 05:11:18 2008 +0300 @@ -143,7 +143,9 @@ case SEARCH_UIDSET: return seq_range_exists(&arg->value.seqset, rec->uid); case SEARCH_FLAGS: - flags = rec->flags; + /* recent flag shouldn't be set, but indexes from v1.0.x + may contain it. */ + flags = rec->flags & ~MAIL_RECENT; if ((arg->value.flags & MAIL_RECENT) != 0 && index_mailbox_is_recent(ctx->ibox, rec->uid)) flags |= MAIL_RECENT;
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/lib-storage/index/index-sort-private.h Mon Jun 09 05:11:18 2008 +0300 @@ -0,0 +1,33 @@ +#ifndef INDEX_SORT_PRIVATE_H +#define INDEX_SORT_PRIVATE_H + +#include "index-sort.h" + +struct mail_search_sort_program { + struct mailbox_transaction_context *t; + enum mail_sort_type sort_program[MAX_SORT_PROGRAM_SIZE]; + struct mail *temp_mail; + + void (*sort_list_add)(struct mail_search_sort_program *program, + struct mail *mail); + void (*sort_list_finish)(struct mail_search_sort_program *program); + void *context; + + ARRAY_TYPE(uint32_t) seqs; + unsigned int iter_idx; + + unsigned int reverse:1; +}; + +int index_sort_header_get(struct mail *mail, uint32_t seq, + enum mail_sort_type sort_type, string_t *dest); +int index_sort_node_cmp_type(struct mail *mail, + const enum mail_sort_type *sort_program, + uint32_t seq1, uint32_t seq2); + +void index_sort_list_init_string(struct mail_search_sort_program *program); +void index_sort_list_add_string(struct mail_search_sort_program *program, + struct mail *mail); +void index_sort_list_finish_string(struct mail_search_sort_program *program); + +#endif
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/lib-storage/index/index-sort-string.c Mon Jun 09 05:11:18 2008 +0300 @@ -0,0 +1,800 @@ +/* Copyright (c) 2006-2008 Dovecot authors, see the included COPYING file */ + +/* The idea is that we use 32bit integers for string sort IDs which specifiy + the sort order for primary sort condition. The whole 32bit integer space is + used and whenever adding a string, the available space is halved and the new + ID is added in the middle. For example if we add one mail the first time, it + gets ID 2^31. If we then add two mails which are sorted before the first + one, they get IDs 2^31/3 and 2^31/3*2. Once we run out of the available + space between IDs, more space is made by renumbering some IDs. +*/ +#include "lib.h" +#include "array.h" +#include "str.h" +#include "index-storage.h" +#include "index-sort-private.h" + +#include <stdlib.h> + +struct mail_sort_node { + uint32_t seq:29; + uint32_t wanted:1; + uint32_t no_update:1; + uint32_t sort_id_changed:1; + uint32_t sort_id; +}; +ARRAY_DEFINE_TYPE(mail_sort_node, struct mail_sort_node); + +struct sort_string_context { + struct mail_search_sort_program *program; + + ARRAY_TYPE(mail_sort_node) zero_nodes, nonzero_nodes, sorted_nodes; + const char **sort_strings; + pool_t sort_string_pool; + unsigned int first_missing_sort_id_idx; + + uint32_t ext_id, last_seq, highest_reset_id; + uint32_t lowest_nonexpunged_zero; + + unsigned int regetting:1; + unsigned int have_all_wanted:1; + unsigned int no_writing:1; +}; + +static char expunged_msg; +static struct sort_string_context *static_zero_cmp_context; +static struct mail_search_sort_program *static_sort_node_cmp_context; + +static void index_sort_node_add(struct sort_string_context *ctx, + struct mail_sort_node *node); + +void index_sort_list_init_string(struct mail_search_sort_program *program) +{ + struct index_mailbox *ibox = (struct index_mailbox *)program->t->box; + struct sort_string_context *ctx; + const char *name; + + switch (program->sort_program[0] & MAIL_SORT_MASK) { + case MAIL_SORT_CC: + name = "sort-c"; + break; + case MAIL_SORT_FROM: + name = "sort-f"; + break; + case MAIL_SORT_SUBJECT: + name = "sort-s"; + break; + case MAIL_SORT_TO: + name = "sort-t"; + break; + default: + i_unreached(); + } + + program->context = ctx = i_new(struct sort_string_context, 1); + ctx->program = program; + ctx->ext_id = mail_index_ext_register(ibox->index, name, 0, + sizeof(uint32_t), + sizeof(uint32_t)); + i_array_init(&ctx->zero_nodes, 128); + i_array_init(&ctx->nonzero_nodes, 128); +} + +static void index_sort_generate_seqs(struct sort_string_context *ctx) +{ + struct mail_sort_node *nodes, *nodes2; + unsigned int i, j, count, count2; + uint32_t seq; + + nodes = array_get_modifiable(&ctx->nonzero_nodes, &count); + nodes2 = array_get_modifiable(&ctx->zero_nodes, &count2); + + if (!array_is_created(&ctx->program->seqs)) + i_array_init(&ctx->program->seqs, count + count2); + else + array_clear(&ctx->program->seqs); + + for (i = j = 0;;) { + if (i < count && j < count2) { + if (nodes[i].seq < nodes2[j].seq) + seq = nodes[i++].seq; + else + seq = nodes2[j++].seq; + } else if (i < count) { + seq = nodes[i++].seq; + } else if (j < count2) { + seq = nodes2[j++].seq; + } else { + break; + } + array_append(&ctx->program->seqs, &seq, 1); + } +} + +static void index_sort_reget_sort_ids(struct sort_string_context *ctx) +{ + struct mail_sort_node node; + const uint32_t *seqs; + unsigned int i, count; + + i_assert(!ctx->regetting); + ctx->regetting = TRUE; + + index_sort_generate_seqs(ctx); + array_clear(&ctx->zero_nodes); + array_clear(&ctx->nonzero_nodes); + + memset(&node, 0, sizeof(node)); + node.wanted = TRUE; + seqs = array_get(&ctx->program->seqs, &count); + for (i = 0; i < count; i++) { + node.seq = seqs[i]; + index_sort_node_add(ctx, &node); + } + ctx->regetting = FALSE; +} + +static void index_sort_node_add(struct sort_string_context *ctx, + struct mail_sort_node *node) +{ + struct index_transaction_context *t = + (struct index_transaction_context *)ctx->program->t; + struct mail_index_map *map; + const void *data; + uint32_t reset_id; + bool expunged; + + mail_index_lookup_ext_full(t->trans_view, node->seq, + ctx->ext_id, &map, &data, &expunged); + if (expunged) { + /* we don't want to update expunged messages' sort IDs */ + node->no_update = TRUE; + /* we can't trust expunged messages' sort IDs. they might be + valid, but it's also possible that sort IDs were updated + and the expunged messages' sort IDs became invalid. we could + use sort ID if we could know the extension's reset_id at the + time of the expunge so we could compare it to + highest_reset_id, but this isn't currently possible. */ + node->sort_id = 0; + } else { + node->sort_id = data == NULL ? 0 : *(const uint32_t *)data; + if (node->sort_id == 0) { + if (ctx->lowest_nonexpunged_zero > node->seq || + ctx->lowest_nonexpunged_zero == 0) + ctx->lowest_nonexpunged_zero = node->seq; + } else { + i_assert(ctx->lowest_nonexpunged_zero == 0 || + ctx->lowest_nonexpunged_zero > node->seq); + } + } + + if (node->sort_id != 0) { + /* if reset ID increases, lookup all existing messages' sort + IDs again. if it decreases, ignore the sort ID. */ + if (!mail_index_ext_get_reset_id(t->trans_view, map, + ctx->ext_id, &reset_id)) + reset_id = 0; + if (reset_id != ctx->highest_reset_id) { + if (reset_id < ctx->highest_reset_id) { + i_assert(expunged); + node->sort_id = 0; + } else if (ctx->have_all_wanted) { + /* a bit late to start changing the reset_id. + the node lists aren't ordered by sequence + anymore. */ + node->sort_id = 0; + ctx->no_writing = TRUE; + } else { + ctx->highest_reset_id = reset_id; + index_sort_reget_sort_ids(ctx); + } + } + } + + if (node->sort_id == 0) + array_append(&ctx->zero_nodes, node, 1); + else + array_append(&ctx->nonzero_nodes, node, 1); + if (ctx->last_seq < node->seq) + ctx->last_seq = node->seq; +} + +void index_sort_list_add_string(struct mail_search_sort_program *program, + struct mail *mail) +{ + struct mail_sort_node node; + + memset(&node, 0, sizeof(node)); + node.seq = mail->seq; + node.wanted = TRUE; + + index_sort_node_add(program->context, &node); +} + +static int sort_node_zero_string_cmp(const void *p1, const void *p2) +{ + struct sort_string_context *ctx = static_zero_cmp_context; + const struct mail_sort_node *n1 = p1, *n2 = p2; + int ret; + + ret = strcmp(ctx->sort_strings[n1->seq], ctx->sort_strings[n2->seq]); + if (ret != 0) + return ret; + + return index_sort_node_cmp_type(ctx->program->temp_mail, + ctx->program->sort_program + 1, + n1->seq, n2->seq); +} + +static void index_sort_zeroes(struct sort_string_context *ctx) +{ + struct mail *mail = ctx->program->temp_mail; + enum mail_sort_type sort_type = ctx->program->sort_program[0]; + string_t *str; + pool_t pool; + struct mail_sort_node *nodes; + unsigned int i, count; + + /* first get all the messages' sort strings. although this takes more + memory, it makes error handling easier and probably also helps + CPU caching. */ + ctx->sort_strings = i_new(const char *, ctx->last_seq + 1); + ctx->sort_string_pool = pool = + pool_alloconly_create("sort strings", 1024*64); + str = t_str_new(512); + nodes = array_get_modifiable(&ctx->zero_nodes, &count); + for (i = 0; i < count; i++) { + i_assert(nodes[i].seq <= ctx->last_seq); + + index_sort_header_get(mail, nodes[i].seq, sort_type, str); + ctx->sort_strings[nodes[i].seq] = str_len(str) == 0 ? "" : + p_strdup(pool, str_c(str)); + } + + /* we have all strings, sort nodes based on them */ + static_zero_cmp_context = ctx; + qsort(nodes, count, sizeof(struct mail_sort_node), + sort_node_zero_string_cmp); +} + +static const char * +index_sort_get_expunged_string(struct sort_string_context *ctx, uint32_t idx, + string_t *str) +{ + struct mail *mail = ctx->program->temp_mail; + enum mail_sort_type sort_type = ctx->program->sort_program[0]; + const struct mail_sort_node *nodes; + const char *result = NULL; + unsigned int i, count; + uint32_t sort_id; + + /* Look forwards and backwards to see if there are + identical sort_ids. If we do find them, try to get + their sort string and use it to update the rest. */ + nodes = array_get(&ctx->nonzero_nodes, &count); + sort_id = nodes[idx].sort_id; + /* If previous sort ID is identical and its sort string is set, we can + trust it. If it's expunged, we already verified that there are no + non-expunged messages. */ + if (idx > 0 && nodes[idx-1].sort_id == sort_id && + ctx->sort_strings[nodes[idx].seq] != NULL) + return ctx->sort_strings[nodes[idx].seq]; + + /* Go forwards as long as there are identical sort IDs. If we find one + that's not expunged, update string table for all messages with + identical sort IDs. */ + for (i = idx + 1; i < count; i++) { + if (nodes[i].sort_id != sort_id) + break; + + if (ctx->sort_strings[nodes[i].seq] != NULL) { + /* usually we fill all identical sort_ids and this + shouldn't happen, but we can get here if we skipped + over messages when binary searching */ + result = ctx->sort_strings[nodes[i].seq]; + break; + } + if (index_sort_header_get(mail, nodes[i].seq, + sort_type, str) >= 0) { + result = str_len(str) == 0 ? "" : + p_strdup(ctx->sort_string_pool, str_c(str)); + break; + } + } + if (result == NULL) { + /* unknown */ + return &expunged_msg; + } + + /* fill all identical sort_ids with the same value */ + for (i = idx; i > 0 && nodes[i-1].sort_id == sort_id; i--) ; + for (i = idx; i < count && nodes[i].sort_id == sort_id; i++) + ctx->sort_strings[nodes[i].seq] = result; + return result; +} + +static const char * +index_sort_get_string(struct sort_string_context *ctx, + uint32_t idx, uint32_t seq) +{ + struct mail *mail = ctx->program->temp_mail; + int ret; + + if (ctx->sort_strings[seq] == NULL) T_BEGIN { + string_t *str; + + str = t_str_new(256); + ret = index_sort_header_get(mail, seq, + ctx->program->sort_program[0], str); + if (str_len(str) > 0) { + ctx->sort_strings[seq] = + p_strdup(ctx->sort_string_pool, str_c(str)); + } else if (ret >= 0) { + ctx->sort_strings[seq] = ""; + } else { + ctx->sort_strings[seq] = + index_sort_get_expunged_string(ctx, idx, str); + } + } T_END; + + return ctx->sort_strings[seq]; +} + +static void +index_sort_bsearch(struct sort_string_context *ctx, const char *key, + unsigned int start_idx, unsigned int *idx_r, + const char **prev_str_r) +{ + const struct mail_sort_node *nodes; + const char *str, *str2; + unsigned int idx, left_idx, right_idx, prev; + int ret; + + nodes = array_get_modifiable(&ctx->nonzero_nodes, &right_idx); + idx = left_idx = start_idx; + while (left_idx < right_idx) { + idx = (left_idx + right_idx) / 2; + str = index_sort_get_string(ctx, idx, nodes[idx].seq); + if (str != &expunged_msg) + ret = strcmp(key, str); + else { + /* put expunged messages first */ + ret = 1; + for (prev = idx; prev > 0; ) { + prev--; + str2 = index_sort_get_string(ctx, prev, + nodes[prev].seq); + if (str2 != &expunged_msg) { + ret = strcmp(key, str2); + if (ret <= 0) { + idx = prev; + str = str2; + } + break; + } + } + } + if (ret > 0) + left_idx = idx+1; + else if (ret < 0) + right_idx = idx; + else { + *idx_r = idx + 1; + *prev_str_r = str; + return; + } + } + + if (left_idx > idx) + idx++; + + *idx_r = idx; + if (idx > start_idx) { + prev = idx; + do { + prev--; + str2 = index_sort_get_string(ctx, prev, + nodes[prev].seq); + } while (str2 == &expunged_msg && prev > 0 && + nodes[prev-1].sort_id == nodes[prev].sort_id); + *prev_str_r = str2; + } +} + +static void index_sort_merge(struct sort_string_context *ctx) +{ + struct mail_sort_node *znodes, *nznodes; + const char *zstr, *nzstr, *prev_str; + unsigned int zpos, nzpos, nz_next_pos, zcount, nzcount; + int ret; + + /* both zero_nodes and nonzero_nodes are sorted. we'll now just have + to merge them together. use sorted_nodes as the result array. */ + i_array_init(&ctx->sorted_nodes, array_count(&ctx->nonzero_nodes) + + array_count(&ctx->zero_nodes)); + + znodes = array_get_modifiable(&ctx->zero_nodes, &zcount); + nznodes = array_get_modifiable(&ctx->nonzero_nodes, &nzcount); + + prev_str = NULL; + for (zpos = nzpos = 0; zpos < zcount && nzpos < nzcount; ) { + zstr = ctx->sort_strings[znodes[zpos].seq]; + nzstr = index_sort_get_string(ctx, nzpos, nznodes[nzpos].seq); + + if (nzstr != &expunged_msg) + ret = strcmp(zstr, nzstr); + else if (prev_str != NULL && strcmp(zstr, prev_str) == 0) { + /* identical to previous message, must keep them + together */ + ret = -1; + } else { + /* we can't be yet sure about the order, but future + nznodes may reveal that the znode must be added + later. if future nznodes don't reveal that, we have + no idea about these nodes' order. so just always + put the expunged message first. */ + ret = 1; + } + + if (ret <= 0) { + array_append(&ctx->sorted_nodes, &znodes[zpos], 1); + prev_str = zstr; + zpos++; + } else { + array_append(&ctx->sorted_nodes, &nznodes[nzpos], 1); + prev_str = nzstr; + nzpos++; + + /* avoid looking up all existing messages' strings by + binary searching the next zero-node position. don't + bother if it looks like more work than linear + scanning. */ + if (zcount - zpos < (nzcount - nzpos)/2) { + index_sort_bsearch(ctx, zstr, nzpos, + &nz_next_pos, &prev_str); + array_append(&ctx->sorted_nodes, + &nznodes[nzpos], + nz_next_pos - nzpos); + nzpos = nz_next_pos; + } + } + } + /* only one of zero_nodes and nonzero_nodes can be non-empty now */ + for (; zpos < zcount; zpos++) + array_append(&ctx->sorted_nodes, &znodes[zpos], 1); + for (; nzpos < nzcount; nzpos++) + array_append(&ctx->sorted_nodes, &nznodes[nzpos], 1); + + /* future index_sort_get_string() calls use ctx->nonzero_nodes, but we + use only ctx->sorted_nodes. make them identical. */ + array_free(&ctx->nonzero_nodes); + ctx->nonzero_nodes = ctx->sorted_nodes; +} + +static int +index_sort_add_ids_range(struct sort_string_context *ctx, + unsigned int left_idx, unsigned int right_idx) +{ + + struct mail_sort_node *nodes; + unsigned int i, count, rightmost_idx, skip; + const char *left_str = NULL, *right_str = NULL, *str; + uint32_t left_sort_id, right_sort_id, diff; + bool no_left_str = FALSE, no_right_str = FALSE; + int ret; + + nodes = array_get_modifiable(&ctx->sorted_nodes, &count); + rightmost_idx = count - 1; + + /* get the sort IDs from left and right */ + left_sort_id = nodes[left_idx].sort_id; + right_sort_id = nodes[right_idx].sort_id; + /* check if all of them should have the same sort IDs. we don't want + to hit the renumbering code in that situation. */ + if (left_sort_id == right_sort_id && left_sort_id != 0) { + /* they should all have the same sort ID */ + for (i = left_idx + 1; i < right_idx; i++) { + nodes[i].sort_id = left_sort_id; + nodes[i].sort_id_changed = TRUE; + } + return 0; + } + + if (left_sort_id == 0) { + i_assert(left_idx == 0); + left_sort_id = 1; + } + if (right_sort_id == 0) { + i_assert(right_idx == rightmost_idx); + right_sort_id = (uint32_t)-1; + } + i_assert(left_sort_id <= right_sort_id); + + diff = right_sort_id - left_sort_id; + while (diff / (right_idx-left_idx + 2) == 0) { + /* we most likely don't have enough space. we have to + renumber some of the existing sort IDs. do this by + widening the area we're giving sort IDs. */ + if (left_idx > 0) { + left_sort_id = nodes[--left_idx].sort_id; + if (left_sort_id == 0) { + i_assert(left_idx == 0); + left_sort_id = 1; + } + } + + while (right_idx < rightmost_idx) { + right_idx++; + if (nodes[right_idx].sort_id > right_sort_id) + break; + } + right_sort_id = nodes[right_idx].sort_id; + if (right_sort_id == 0) { + i_assert(right_idx == rightmost_idx); + right_sort_id = (uint32_t)-1; + } + i_assert(left_sort_id < right_sort_id); + + if (diff == right_sort_id - left_sort_id) { + /* we did nothing, but there's still not enough space. + have to renumber the leftmost/rightmost node(s) */ + i_assert(left_idx == 0 && right_idx == rightmost_idx); + if (left_sort_id > 1) { + left_sort_id = 1; + no_left_str = TRUE; + } else { + i_assert(right_sort_id != (uint32_t)-1); + right_sort_id = (uint32_t)-1; + no_right_str = TRUE; + } + } + diff = right_sort_id - left_sort_id; + } + + if (nodes[left_idx].sort_id != 0 && !no_left_str) { + left_str = index_sort_get_string(ctx, left_idx, + nodes[left_idx].seq); + if (left_str == &expunged_msg) { + /* not equivalent with any message */ + left_str = NULL; + } + left_idx++; + } + if (nodes[right_idx].sort_id != 0 && !no_right_str) { + right_str = index_sort_get_string(ctx, right_idx, + nodes[right_idx].seq); + if (right_str == &expunged_msg) { + /* not equivalent with any message */ + right_str = NULL; + } + right_idx--; + } + i_assert(left_idx <= right_idx); + + /* give (new) sort IDs to all nodes in left_idx..right_idx range. + divide the available space so that each message gets an equal sized + share. some messages' sort strings may be equivalent, so give them + the same sort IDs. */ + for (i = left_idx; i <= right_idx; i++) { + str = index_sort_get_string(ctx, i, nodes[i].seq); + if (str == &expunged_msg) { + /* it doesn't really matter what we give to this + message, since it's only temporary and we don't + know its correct position anyway. so let's assume + it's equivalent to previous message. */ + nodes[i].sort_id = left_sort_id; + continue; + } + + ret = left_str == NULL ? 1 : strcmp(str, left_str); + if (ret <= 0) { + if (ret < 0) { + /* broken sort_ids */ + return -1; + } + nodes[i].sort_id = left_sort_id; + } else if (right_str != NULL && strcmp(str, right_str) == 0) { + /* the rest of the sort IDs should be the same */ + nodes[i].sort_id = right_sort_id; + left_sort_id = right_sort_id; + } else { + /* divide the available space equally. leave the same + sized space also between the first and the last + messages */ + skip = (right_sort_id - left_sort_id) / + (right_idx - i + 2); + i_assert(skip > 0); + left_sort_id += skip; + i_assert(left_sort_id < right_sort_id); + + nodes[i].sort_id = left_sort_id; + left_str = str; + } + nodes[i].sort_id_changed = TRUE; + } + return right_str == NULL || strcmp(str, right_str) < 0 ? 0 : -1; +} + +static int +index_sort_add_sort_ids(struct sort_string_context *ctx) +{ + const struct mail_sort_node *nodes; + unsigned int i, left_idx, right_idx, count; + + nodes = array_get(&ctx->sorted_nodes, &count); + for (i = 0; i < count; i++) { + if (nodes[i].sort_id != 0) + continue; + + /* get the range for all sort_id=0 nodes. include the nodes + left and right of the range as well */ + for (right_idx = i + 1; right_idx < count; right_idx++) { + if (nodes[right_idx].sort_id != 0) + break; + } + if (right_idx == count) + right_idx--; + left_idx = i == 0 ? 0 : i - 1; + if (index_sort_add_ids_range(ctx, left_idx, right_idx) < 0) + return -1; + } + return 0; +} + +static void index_sort_write_changed_sort_ids(struct sort_string_context *ctx) +{ + struct index_transaction_context *t = + (struct index_transaction_context *)ctx->program->t; + uint32_t ext_id = ctx->ext_id; + const struct mail_sort_node *nodes; + unsigned int i, count; + + if (ctx->no_writing) { + /* our reset_id is already stale - don't even bother + trying to write */ + return; + } + + mail_index_ext_reset_inc(t->trans, ext_id, ctx->highest_reset_id, FALSE); + + /* add the missing sort IDs to index */ + nodes = array_get_modifiable(&ctx->sorted_nodes, &count); + for (i = 0; i < count; i++) { + i_assert(nodes[i].sort_id != 0); + if (!nodes[i].sort_id_changed || nodes[i].no_update) + continue; + + mail_index_update_ext(t->trans, nodes[i].seq, ext_id, + &nodes[i].sort_id, NULL); + } +} + +static int sort_node_cmp(const void *p1, const void *p2) +{ + struct mail_search_sort_program *program = static_sort_node_cmp_context; + const struct mail_sort_node *n1 = p1, *n2 = p2; + + if (n1->sort_id < n2->sort_id) + return -1; + if (n1->sort_id > n2->sort_id) + return 1; + + return index_sort_node_cmp_type(program->temp_mail, + program->sort_program + 1, + n1->seq, n2->seq); +} + +static void index_sort_add_missing(struct sort_string_context *ctx) +{ + struct mail_sort_node node; + const uint32_t *seqs; + unsigned int i, count; + uint32_t seq, next_seq; + + ctx->have_all_wanted = TRUE; + + seqs = array_get(&ctx->program->seqs, &count); + for (i = 0, next_seq = 1; i < count; i++) { + if (seqs[i] == next_seq) + next_seq++; + else { + i_assert(next_seq < seqs[i]); + for (seq = next_seq; seq < seqs[i]; seq++) { + memset(&node, 0, sizeof(node)); + node.seq = seq; + index_sort_node_add(ctx, &node); + } + next_seq = seqs[i] + 1; + } + } + + if (ctx->lowest_nonexpunged_zero == 0) { + /* we're handling only expunged zeros. if it causes us to + renumber some existing sort IDs, don't save them. */ + ctx->no_writing = TRUE; + } +} + +static void index_sort_list_reset_broken(struct sort_string_context *ctx) +{ + struct mailbox *box = ctx->program->t->box; + struct mail_sort_node *nodes; + unsigned int i, count; + + mail_storage_set_critical(box->storage, + "Sort IDs %u broken in mailbox %s, reseting", + ctx->ext_id, box->name); + + array_clear(&ctx->zero_nodes); + array_append_array(&ctx->zero_nodes, + &ctx->nonzero_nodes); + array_clear(&ctx->nonzero_nodes); + + nodes = array_get_modifiable(&ctx->zero_nodes, &count); + for (i = 0; i < count; i++) + nodes[i].sort_id = 0; +} + +void index_sort_list_finish_string(struct mail_search_sort_program *program) +{ + struct sort_string_context *ctx = program->context; + struct mail_sort_node *nodes; + unsigned int i, count; + uint32_t seq; + + nodes = array_get_modifiable(&ctx->nonzero_nodes, &count); + + static_sort_node_cmp_context = program; + if (array_count(&ctx->zero_nodes) == 0) { + /* fast path: we have all sort IDs */ + qsort(nodes, count, sizeof(struct mail_sort_node), + sort_node_cmp); + + i_array_init(&program->seqs, count); + for (i = 0; i < count; i++) { + seq = nodes[i].seq; + array_append(&program->seqs, &seq, 1); + } + array_free(&ctx->nonzero_nodes); + } else { + /* we have to add some sort IDs. we'll do this for all + messages, so first remember what messages we wanted + to know about. */ + index_sort_generate_seqs(ctx); + /* add messages not in seqs list */ + index_sort_add_missing(ctx); + /* sort all messages with sort IDs */ + nodes = array_get_modifiable(&ctx->nonzero_nodes, &count); + qsort(nodes, count, sizeof(struct mail_sort_node), + sort_node_cmp); + for (;;) { + /* sort all messages without sort IDs */ + index_sort_zeroes(ctx); + /* merge zero and non-zero arrays into sorted_nodes */ + index_sort_merge(ctx); + /* give sort IDs to messages missing them */ + if (index_sort_add_sort_ids(ctx) == 0) + break; + + /* broken, try again */ + index_sort_list_reset_broken(ctx); + } + index_sort_write_changed_sort_ids(ctx); + + nodes = array_get_modifiable(&ctx->sorted_nodes, &count); + array_clear(&program->seqs); + for (i = 0; i < count; i++) { + if (nodes[i].wanted) { + seq = nodes[i].seq; + array_append(&program->seqs, &seq, 1); + } + } + pool_unref(&ctx->sort_string_pool); + i_free(ctx->sort_strings); + array_free(&ctx->sorted_nodes); + /* NOTE: we already freed nonzero_nodes and made it point to + sorted_nodes. */ + } + + array_free(&ctx->zero_nodes); +}
--- a/src/lib-storage/index/index-sort.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/index-sort.c Mon Jun 09 05:11:18 2008 +0300 @@ -1,56 +1,27 @@ /* Copyright (c) 2006-2008 Dovecot authors, see the included COPYING file */ -/* The idea in here is that we use a 32bit integer (sort ID) which specifies - the sort order for primary sort condition. With fixed size fields (time, - size) we use the field itself as the sort ID. They can be looked up fast - enough from cache file, so we don't add them to index file. - - Strings can't be used as sort IDs directly. The way they're currently - handled is that the whole 32bit integer space is used for them and whenever - adding a string, the available space is halved and the new ID is added in - the middle. For example if we add one mail the first time, it gets ID - 2^31. If we then add two mails which are sorted before the first one, they - get IDs 2^31/3 and 2^31/3*2. Once we run out of the available space between - IDs, a large amount of the IDs are renumbered. -*/ - #include "lib.h" #include "array.h" -#include "bsearch-insert-pos.h" #include "str.h" #include "unichar.h" #include "message-address.h" #include "imap-base-subject.h" #include "index-storage.h" -#include "index-sort.h" +#include "index-sort-private.h" #include <stdlib.h> -#define RENUMBER_SPACE 100 - -struct mail_sort_node { +struct mail_sort_node_date { uint32_t seq; - uint32_t sort_id; + time_t date; }; -ARRAY_DEFINE_TYPE(mail_sort_node, struct mail_sort_node); - -struct mail_search_sort_program { - struct mailbox_transaction_context *t; - enum mail_sort_type sort_program[MAX_SORT_PROGRAM_SIZE]; - struct mail *temp_mail; +ARRAY_DEFINE_TYPE(mail_sort_node_date, struct mail_sort_node_date); - ARRAY_TYPE(mail_sort_node) nodes, all_nodes; - const struct mail_sort_node *nodes_ptr; - unsigned int nodes_count, iter_idx; - - uint32_t ext_id; - unsigned int first_missing_sort_id_idx; - uint32_t (*get_sort_id)(struct mail *); - - unsigned int reverse:1; - unsigned int sort_ids_added:1; - unsigned int missing_sort_ids:1; +struct mail_sort_node_size { + uint32_t seq; + uoff_t size; }; +ARRAY_DEFINE_TYPE(mail_sort_node_size, struct mail_sort_node_size); struct sort_cmp_context { struct mail_search_sort_program *program; @@ -59,17 +30,154 @@ static struct sort_cmp_context static_node_cmp_context; -static uint32_t sort_get_arrival(struct mail *mail); -static uint32_t sort_get_date(struct mail *mail); -static uint32_t sort_get_size(struct mail *mail); +static void +index_sort_list_add_arrival(struct mail_search_sort_program *program, + struct mail *mail) +{ + ARRAY_TYPE(mail_sort_node_date) *nodes = program->context; + struct mail_sort_node_date *node; + + node = array_append_space(nodes); + node->seq = mail->seq; + if (mail_get_received_date(mail, &node->date) < 0) + node->date = 0; +} + +static void +index_sort_list_add_date(struct mail_search_sort_program *program, + struct mail *mail) +{ + ARRAY_TYPE(mail_sort_node_date) *nodes = program->context; + struct mail_sort_node_date *node; + + node = array_append_space(nodes); + node->seq = mail->seq; + if (mail_get_date(mail, &node->date, NULL) < 0) + node->date = 0; + else if (node->date == 0) { + if (mail_get_received_date(mail, &node->date) < 0) + node->date = 0; + } +} + +static void +index_sort_list_add_size(struct mail_search_sort_program *program, + struct mail *mail) +{ + ARRAY_TYPE(mail_sort_node_size) *nodes = program->context; + struct mail_sort_node_size *node; + + node = array_append_space(nodes); + node->seq = mail->seq; + if (mail_get_virtual_size(mail, &node->size) < 0) + node->size = 0; +} + +void index_sort_list_add(struct mail_search_sort_program *program, + struct mail *mail) +{ + i_assert(mail->transaction == program->t); + + program->sort_list_add(program, mail); +} + +static int sort_node_date_cmp(const void *p1, const void *p2) +{ + struct sort_cmp_context *ctx = &static_node_cmp_context; + const struct mail_sort_node_date *n1 = p1, *n2 = p2; + + if (n1->date < n2->date) + return -1; + if (n2->date > n2->date) + return 1; + + return index_sort_node_cmp_type(ctx->mail, + ctx->program->sort_program + 1, + n1->seq, n2->seq); +} + +static void +index_sort_list_finish_date(struct mail_search_sort_program *program) +{ + ARRAY_TYPE(mail_sort_node_date) *nodes = program->context; + struct mail_sort_node_date *date_nodes; + unsigned int count; + + date_nodes = array_get_modifiable(nodes, &count); + qsort(date_nodes, count, sizeof(struct mail_sort_node_date), + sort_node_date_cmp); + memcpy(&program->seqs, nodes, sizeof(program->seqs)); + i_free(nodes); + program->context = NULL; +} + +static int sort_node_size_cmp(const void *p1, const void *p2) +{ + struct sort_cmp_context *ctx = &static_node_cmp_context; + const struct mail_sort_node_size *n1 = p1, *n2 = p2; + + if (n1->size < n2->size) + return -1; + if (n2->size > n2->size) + return 1; + + return index_sort_node_cmp_type(ctx->mail, + ctx->program->sort_program + 1, + n1->seq, n2->seq); +} + +static void +index_sort_list_finish_size(struct mail_search_sort_program *program) +{ + ARRAY_TYPE(mail_sort_node_size) *nodes = program->context; + struct mail_sort_node_size *size_nodes; + unsigned int count; + + size_nodes = array_get_modifiable(nodes, &count); + qsort(size_nodes, count, sizeof(struct mail_sort_node_size), + sort_node_size_cmp); + memcpy(&program->seqs, nodes, sizeof(program->seqs)); + i_free(nodes); + program->context = NULL; +} + +void index_sort_list_finish(struct mail_search_sort_program *program) +{ + memset(&static_node_cmp_context, 0, sizeof(static_node_cmp_context)); + static_node_cmp_context.program = program; + static_node_cmp_context.mail = program->temp_mail; + + program->sort_list_finish(program); + + if (program->reverse) + program->iter_idx = array_count(&program->seqs); +} + +bool index_sort_list_next(struct mail_search_sort_program *program, + struct mail *mail) +{ + const uint32_t *seqp; + + if (!program->reverse) { + if (program->iter_idx == array_count(&program->seqs)) + return FALSE; + + seqp = array_idx(&program->seqs, program->iter_idx++); + } else { + if (program->iter_idx == 0) + return FALSE; + + seqp = array_idx(&program->seqs, --program->iter_idx); + } + mail_set_seq(mail, *seqp); + return TRUE; +} struct mail_search_sort_program * index_sort_program_init(struct mailbox_transaction_context *t, const enum mail_sort_type *sort_program) { - struct index_mailbox *ibox = (struct index_mailbox *)t->box; struct mail_search_sort_program *program; - const char *name = NULL; unsigned int i; if (sort_program == NULL || sort_program[0] == MAIL_SORT_END) @@ -79,7 +187,6 @@ program = i_new(struct mail_search_sort_program, 1); program->t = t; program->temp_mail = mail_alloc(t, 0, NULL); - i_array_init(&program->nodes, 64); /* primary reversion isn't stored to sort_program. we handle it by iterating backwards at the end. */ @@ -95,32 +202,42 @@ switch (program->sort_program[0] & MAIL_SORT_MASK) { case MAIL_SORT_ARRIVAL: - program->get_sort_id = sort_get_arrival; + case MAIL_SORT_DATE: { + ARRAY_TYPE(mail_sort_node_date) *nodes; + + nodes = i_malloc(sizeof(*nodes)); + i_array_init(nodes, 128); + + if ((program->sort_program[0] & + MAIL_SORT_MASK) == MAIL_SORT_ARRIVAL) + program->sort_list_add = index_sort_list_add_arrival; + else + program->sort_list_add = index_sort_list_add_date; + program->sort_list_finish = index_sort_list_finish_date; + program->context = nodes; break; - case MAIL_SORT_DATE: - program->get_sort_id = sort_get_date; + } + case MAIL_SORT_SIZE: { + ARRAY_TYPE(mail_sort_node_size) *nodes; + + nodes = i_malloc(sizeof(*nodes)); + i_array_init(nodes, 128); + program->sort_list_add = index_sort_list_add_size; + program->sort_list_finish = index_sort_list_finish_size; + program->context = nodes; break; - case MAIL_SORT_SIZE: - program->get_sort_id = sort_get_size; - break; + } case MAIL_SORT_CC: - name = "sort-c"; - break; case MAIL_SORT_FROM: - name = "sort-f"; - break; case MAIL_SORT_SUBJECT: - name = "sort-s"; - break; case MAIL_SORT_TO: - name = "sort-t"; + program->sort_list_add = index_sort_list_add_string; + program->sort_list_finish = index_sort_list_finish_string; + index_sort_list_init_string(program); break; default: i_unreached(); } - program->ext_id = name == NULL ? (uint32_t)-1 : - mail_index_ext_register(ibox->index, name, 0, - sizeof(uint32_t), sizeof(uint32_t)); return program; } @@ -130,106 +247,70 @@ *_program = NULL; mail_free(&program->temp_mail); - array_free(&program->nodes); + array_free(&program->seqs); i_free(program); } -static const char *get_first_mailbox(struct mail *mail, const char *header) +static int +get_first_mailbox(struct mail *mail, const char *header, const char **mailbox_r) { struct message_address *addr; const char *str; + int ret; - if (mail_get_first_header_utf8(mail, header, &str) <= 0) - return ""; + if ((ret = mail_get_first_header_utf8(mail, header, &str)) <= 0) { + *mailbox_r = ""; + return ret; + } addr = message_address_parse(pool_datastack_create(), (const unsigned char *)str, strlen(str), 1, TRUE); - return addr != NULL ? addr->mailbox : ""; + *mailbox_r = addr != NULL ? addr->mailbox : ""; + return 0; } -static void -sort_header_get(string_t *dest, enum mail_sort_type sort_type, - struct mail *mail, uint32_t seq) +int index_sort_header_get(struct mail *mail, uint32_t seq, + enum mail_sort_type sort_type, string_t *dest) { const char *str; + int ret; mail_set_seq(mail, seq); + str_truncate(dest, 0); + switch (sort_type & MAIL_SORT_MASK) { case MAIL_SORT_SUBJECT: - if (mail_get_first_header(mail, "Subject", &str) <= 0) - return; + if ((ret = mail_get_first_header(mail, "Subject", &str)) <= 0) + return ret; str = imap_get_base_subject_cased(pool_datastack_create(), str, NULL); str_append(dest, str); - return; + return 0; case MAIL_SORT_CC: - str = get_first_mailbox(mail, "Cc"); + ret = get_first_mailbox(mail, "Cc", &str); break; case MAIL_SORT_FROM: - str = get_first_mailbox(mail, "From"); + ret = get_first_mailbox(mail, "From", &str); break; case MAIL_SORT_TO: - str = get_first_mailbox(mail, "To"); + ret = get_first_mailbox(mail, "To", &str); break; default: i_unreached(); } (void)uni_utf8_to_decomposed_titlecase(str, (size_t)-1, dest); -} - -static uint32_t sort_get_arrival(struct mail *mail) -{ - time_t t; - - if (mail_get_received_date(mail, &t) < 0) - t = 0; - - i_assert(t != (time_t)-1); - /* FIXME: truncation isn't good.. */ - return t <= 0 ? 1 : - ((uint64_t)t >= (uint32_t)-1 ? (uint32_t)-1 : (uint32_t)t + 1); + return ret; } -static uint32_t sort_get_date(struct mail *mail) -{ - time_t t; - - if (mail_get_date(mail, &t, NULL) < 0) - t = 0; - if (t == 0) { - if (mail_get_received_date(mail, &t) < 0) - return 1; - } - i_assert(t != (time_t)-1); - /* FIXME: truncation isn't good.. */ - return t <= 0 ? 1 : - ((uint64_t)t >= (uint32_t)-1 ? (uint32_t)-1 : (uint32_t)t + 1); -} - -static uint32_t sort_get_size(struct mail *mail) -{ - uoff_t size; - - if (mail_get_virtual_size(mail, &size) < 0) - return 1; - - /* FIXME: elsewhere we support 64bit message sizes, but here - we support only 32bit sizes.. It's a bit too much trouble - to support 64bit here currently, so until such messages - actually start showing up somewhere, 32bit is enough */ - i_assert(size < (uint32_t)-1); - return size + 1; -} - -static int sort_node_cmp_type(struct sort_cmp_context *ctx, - const enum mail_sort_type *sort_program, - const struct mail_sort_node *n1, - const struct mail_sort_node *n2) +int index_sort_node_cmp_type(struct mail *mail, + const enum mail_sort_type *sort_program, + uint32_t seq1, uint32_t seq2) { enum mail_sort_type sort_type; - uint32_t time1, time2, size1, size2; + time_t time1, time2; + uoff_t size1, size2; int ret = 0; sort_type = *sort_program & MAIL_SORT_MASK; @@ -243,353 +324,71 @@ str1 = t_str_new(256); str2 = t_str_new(256); - sort_header_get(str1, sort_type, ctx->mail, n1->seq); - sort_header_get(str2, sort_type, ctx->mail, n2->seq); + index_sort_header_get(mail, seq1, sort_type, str1); + index_sort_header_get(mail, seq2, sort_type, str2); ret = strcmp(str_c(str1), str_c(str2)); } T_END; break; case MAIL_SORT_ARRIVAL: - mail_set_seq(ctx->mail, n1->seq); - time1 = sort_get_arrival(ctx->mail); + mail_set_seq(mail, seq1); + if (mail_get_received_date(mail, &time1) < 0) + time1 = 0; - mail_set_seq(ctx->mail, n2->seq); - time2 = sort_get_arrival(ctx->mail); + mail_set_seq(mail, seq2); + if (mail_get_received_date(mail, &time2) < 0) + time1 = 0; ret = time1 < time2 ? -1 : (time1 > time2 ? 1 : 0); break; case MAIL_SORT_DATE: - mail_set_seq(ctx->mail, n1->seq); - time1 = sort_get_date(ctx->mail); + mail_set_seq(mail, seq1); + if (mail_get_date(mail, &time1, NULL) < 0) + time1 = 0; + else if (time1 == 0) { + if (mail_get_received_date(mail, &time1) < 0) + time1 = 0; + } - mail_set_seq(ctx->mail, n2->seq); - time2 = sort_get_date(ctx->mail); + mail_set_seq(mail, seq2); + if (mail_get_date(mail, &time2, NULL) < 0) + time2 = 0; + else if (time2 == 0) { + if (mail_get_received_date(mail, &time2) < 0) + time2 = 0; + } ret = time1 < time2 ? -1 : (time1 > time2 ? 1 : 0); break; case MAIL_SORT_SIZE: - mail_set_seq(ctx->mail, n1->seq); - size1 = sort_get_size(ctx->mail); + mail_set_seq(mail, seq1); + if (mail_get_virtual_size(mail, &size1) < 0) + size1 = 0; - mail_set_seq(ctx->mail, n2->seq); - size2 = sort_get_size(ctx->mail); + mail_set_seq(mail, seq2); + if (mail_get_virtual_size(mail, &size2) < 0) + size2 = 0; ret = size1 < size2 ? -1 : (size1 > size2 ? 1 : 0); break; case MAIL_SORT_END: - return n1->seq < n2->seq ? -1 : - (n1->seq > n2->seq ? 1 : 0); + return seq1 < seq2 ? -1 : + (seq1 > seq2 ? 1 : 0); case MAIL_SORT_MASK: case MAIL_SORT_FLAG_REVERSE: i_unreached(); } - if (ret == 0) - return sort_node_cmp_type(ctx, sort_program+1, n1, n2); + if (ret == 0) { + return index_sort_node_cmp_type(mail, sort_program+1, + seq1, seq2); + } /* primary reversion isn't in sort_program */ if ((*sort_program & MAIL_SORT_FLAG_REVERSE) != 0) ret = ret < 0 ? 1 : -1; return ret; } - -static int sort_node_cmp(const void *p1, const void *p2) -{ - struct sort_cmp_context *ctx = &static_node_cmp_context; - const struct mail_sort_node *n1 = p1, *n2 = p2; - - if (n1->sort_id < n2->sort_id) - return -1; - if (n1->sort_id > n2->sort_id) - return 1; - - return sort_node_cmp_type(ctx, ctx->program->sort_program + 1, n1, n2); -} - -static int sort_node_cmp_nozero_sort_id(const void *p1, const void *p2) -{ - struct sort_cmp_context *ctx = &static_node_cmp_context; - const struct mail_sort_node *n1 = p1, *n2 = p2; - const enum mail_sort_type *sort_program; - - /* Use sort IDs only if both have them */ - if (n1->sort_id != 0 && n2->sort_id != 0) { - if (n1->sort_id < n2->sort_id) - return -1; - if (n1->sort_id > n2->sort_id) - return 1; - sort_program = ctx->program->sort_program + 1; - } else { - sort_program = ctx->program->sort_program; - } - - return sort_node_cmp_type(ctx, sort_program, n1, n2); -} - -static void -index_sort_add_ids_range(struct mail_search_sort_program *program, - struct mail *mail, unsigned int left_idx, - unsigned int right_idx) -{ - struct mail_sort_node *nodes; - unsigned int i, count, rightmost_idx; - uint32_t left_sort_id, right_sort_id; - string_t *right_str, *left_str, *str; - unsigned int skip; - bool have_right_sort_id = FALSE; - - nodes = array_get_modifiable(&program->all_nodes, &count); - rightmost_idx = count - 1; - - /* get the sort IDs from left and right */ - left_sort_id = left_idx == 0 ? 1 : nodes[left_idx].sort_id; - right_sort_id = right_idx == rightmost_idx ? (uint32_t)-1 : - nodes[right_idx].sort_id; - i_assert(left_sort_id != 0 && right_sort_id != 0); - - while ((right_sort_id - left_sort_id) / (right_idx-left_idx + 2) == 0) { - /* we most likely don't have enough space. we have to - renumber some of the existing sort IDs. do this by - widening the area we're giving sort IDs. */ - if (left_idx > 0) { - left_idx--; - left_sort_id = left_idx == 0 ? 1 : - nodes[left_idx].sort_id; - i_assert(left_sort_id != 0); - } - - while (right_idx < rightmost_idx) { - if (nodes[++right_idx].sort_id != 0) - break; - } - right_sort_id = right_idx == rightmost_idx ? (uint32_t)-1 : - nodes[right_idx].sort_id; - i_assert(left_sort_id < right_sort_id); - } - - left_str = t_str_new(256); - right_str = t_str_new(256); - if (nodes[left_idx].sort_id != 0) { - sort_header_get(left_str, program->sort_program[0], mail, - nodes[left_idx].seq); - left_idx++; - } - if (nodes[right_idx].sort_id != 0) { - have_right_sort_id = TRUE; - sort_header_get(right_str, program->sort_program[0], mail, - nodes[right_idx].seq); - right_idx--; - } - - /* give (new) sort IDs to all nodes in left_idx..right_idx range. - divide the available space so that each messagets an equal sized - share. some messages' sort strings may be equivalent, so give them - the same sort IDs. */ - str = str_new(default_pool, 256); - for (i = left_idx; i <= right_idx; i++) { - T_BEGIN { - sort_header_get(str, program->sort_program[0], mail, - nodes[i].seq); - } T_END; - - if (left_idx != 0 && str_equals(str, left_str)) - nodes[i].sort_id = left_sort_id; - else if (have_right_sort_id && str_equals(str, right_str)) { - /* the rest of the sort IDs should be the same */ - nodes[i].sort_id = right_sort_id; - } else { - /* divide the available space equally. leave the same - sized space also between the first and the last - messages */ - skip = (right_sort_id - left_sort_id) / - (right_idx - i + 2); - i_assert(skip > 0); - left_sort_id += skip; - nodes[i].sort_id = left_sort_id; - - str_truncate(left_str, 0); - str_append_str(left_str, str); - } - } - str_free(&str); -} - -static void -index_sort_add_ids(struct mail_search_sort_program *program, struct mail *mail) -{ - const struct mail_sort_node *nodes; - unsigned int i, left_idx = 0, right_idx, count; - - nodes = array_get(&program->all_nodes, &count); - for (i = 0; i < count; i++) { - if (nodes[i].sort_id != 0) - continue; - - /* get the range for all sort_id=0 nodes. include the nodes - left and right of the range as well */ - for (right_idx = i + 1; right_idx < count; right_idx++) { - if (nodes[right_idx].sort_id != 0) - break; - } - if (right_idx == count) - right_idx--; - left_idx = i == 0 ? 0 : i - 1; - T_BEGIN { - index_sort_add_ids_range(program, mail, - left_idx, right_idx); - } T_END; - } -} - -static void -index_sort_add_string_sort_ids(struct mail_search_sort_program *program, - uint32_t last_seq) -{ - ARRAY_TYPE(mail_sort_node) seq_nodes_arr; - struct mail_sort_node *nodes, node, *seq_nodes; - unsigned int i, count, count2; - - /* insert missing nodes */ - memset(&node, 0, sizeof(node)); - node.seq = array_count(&program->all_nodes) + 1; - for (; node.seq <= last_seq; node.seq++) - array_append(&program->all_nodes, &node, 1); - - /* sort everything. use sort_ids whenever possible */ - nodes = array_get_modifiable(&program->all_nodes, &count); - i_assert(count == last_seq); - qsort(nodes, count, sizeof(struct mail_sort_node), - sort_node_cmp_nozero_sort_id); - - /* we can now build the sort_ids */ - index_sort_add_ids(program, static_node_cmp_context.mail); - - /* @UNSAFE: and finally get the range sorted back by sequence */ - i_array_init(&seq_nodes_arr, count); - (void)array_idx_modifiable(&seq_nodes_arr, count-1); - seq_nodes = array_get_modifiable(&seq_nodes_arr, &count2); - i_assert(count2 == count); - for (i = 0; i < count; i++) - seq_nodes[nodes[i].seq-1] = nodes[i]; - - array_free(&program->all_nodes); - program->all_nodes = seq_nodes_arr; -} - -static void index_sort_build(struct mail_search_sort_program *program) -{ - struct index_transaction_context *t = - (struct index_transaction_context *)program->t; - struct mail_sort_node node, *all_nodes, *nodes; - const void *data; - uint32_t last_seq; - unsigned int seq, i, count, count2; - - /* add messages that have sort_ids. they're always at the beginning - of the mailbox. */ - memset(&node, 0, sizeof(node)); - last_seq = mail_index_view_get_messages_count(t->ibox->view); - i_array_init(&program->all_nodes, last_seq); - for (seq = 1; seq <= last_seq; seq++) { - node.seq = seq; - - mail_index_lookup_ext(t->ibox->view, seq, program->ext_id, - &data, NULL); - node.sort_id = data == NULL ? 0 : *(const uint32_t *)data; - if (node.sort_id == 0) { - /* the rest don't have sort_ids either */ - break; - } - array_append(&program->all_nodes, &node, 1); - } - i_assert(seq <= last_seq); - index_sort_add_string_sort_ids(program, last_seq); - - /* add the missing sort IDs to index */ - all_nodes = array_get_modifiable(&program->all_nodes, &count); - for (; seq <= count; seq++) { - i_assert(all_nodes[seq-1].sort_id != 0); - mail_index_update_ext(t->trans, seq, program->ext_id, - &all_nodes[seq-1].sort_id, NULL); - } - - /* set missing sort_ids to wanted nodes */ - nodes = array_get_modifiable(&program->nodes, &count2); - for (i = program->first_missing_sort_id_idx; i < count2; i++) - nodes[i].sort_id = all_nodes[nodes[i].seq-1].sort_id; - array_free(&program->all_nodes); -} - -void index_sort_list_add(struct mail_search_sort_program *program, - struct mail *mail) -{ - struct index_transaction_context *t = - (struct index_transaction_context *)program->t; - const void *data; - struct mail_sort_node node; - - i_assert(mail->transaction == program->t); - - node.seq = mail->seq; - if (program->ext_id == (uint32_t)-1) { - /* no indexing for this field */ - node.sort_id = program->get_sort_id(mail); - array_append(&program->nodes, &node, 1); - return; - } - - mail_index_lookup_ext(t->trans_view, mail->seq, - program->ext_id, &data, NULL); - node.sort_id = data == NULL ? 0 : *(const uint32_t *)data; - if (node.sort_id == 0 && !program->missing_sort_ids) { - program->missing_sort_ids = TRUE; - program->first_missing_sort_id_idx = - array_count(&program->nodes); - } - array_append(&program->nodes, &node, 1); -} - -void index_sort_list_finish(struct mail_search_sort_program *program) -{ - struct mail_sort_node *nodes; - - memset(&static_node_cmp_context, 0, sizeof(static_node_cmp_context)); - static_node_cmp_context.program = program; - static_node_cmp_context.mail = program->temp_mail; - - if (program->missing_sort_ids) { - i_assert(program->ext_id != (uint32_t)-1); - index_sort_build(program); - } - - nodes = array_get_modifiable(&program->nodes, &program->nodes_count); - qsort(nodes, program->nodes_count, sizeof(struct mail_sort_node), - sort_node_cmp); - - program->nodes_ptr = nodes; - if (program->reverse) - program->iter_idx = program->nodes_count; -} - -bool index_sort_list_next(struct mail_search_sort_program *program, - struct mail *mail) -{ - const struct mail_sort_node *node; - - if (!program->reverse) { - if (program->iter_idx == program->nodes_count) - return FALSE; - - node = &program->nodes_ptr[program->iter_idx++]; - } else { - if (program->iter_idx == 0) - return FALSE; - - node = &program->nodes_ptr[--program->iter_idx]; - } - mail_set_seq(mail, node->seq); - return TRUE; -}
--- a/src/lib-storage/index/index-storage.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/index-storage.h Mon Jun 09 05:11:18 2008 +0300 @@ -123,6 +123,7 @@ uint32_t seq1, uint32_t seq2); bool index_mailbox_is_recent(struct index_mailbox *ibox, uint32_t uid); unsigned int index_mailbox_get_recent_count(struct index_mailbox *ibox); +void index_mailbox_reset_uidvalidity(struct index_mailbox *ibox); void index_mailbox_check_add(struct index_mailbox *ibox, const char *path);
--- a/src/lib-storage/index/index-sync.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/index-sync.c Mon Jun 09 05:11:18 2008 +0300 @@ -49,6 +49,15 @@ seq_range_exists(&ibox->recent_flags, uid); } +void index_mailbox_reset_uidvalidity(struct index_mailbox *ibox) +{ + /* can't trust the currently cached recent flags anymore */ + if (array_is_created(&ibox->recent_flags)) + array_clear(&ibox->recent_flags); + ibox->recent_flags_count = 0; + ibox->recent_flags_prev_uid = 0; +} + unsigned int index_mailbox_get_recent_count(struct index_mailbox *ibox) { const struct mail_index_header *hdr; @@ -95,14 +104,14 @@ static void index_view_sync_recs_get(struct index_mailbox_sync_context *ctx) { - struct mail_index_view_sync_rec sync; + struct mail_index_view_sync_rec sync_rec; uint32_t seq1, seq2; i_array_init(&ctx->flag_updates, 128); if ((ctx->ibox->box.enabled_features & MAILBOX_FEATURE_CONDSTORE) != 0) i_array_init(&ctx->modseq_updates, 32); - while (mail_index_view_sync_next(ctx->sync_ctx, &sync)) { - switch (sync.type) { + while (mail_index_view_sync_next(ctx->sync_ctx, &sync_rec)) { + switch (sync_rec.type) { case MAIL_INDEX_SYNC_TYPE_APPEND: /* not interested */ break; @@ -114,11 +123,11 @@ case MAIL_INDEX_SYNC_TYPE_KEYWORD_REMOVE: case MAIL_INDEX_SYNC_TYPE_KEYWORD_RESET: if (!mail_index_lookup_seq_range(ctx->ibox->view, - sync.uid1, sync.uid2, + sync_rec.uid1, sync_rec.uid2, &seq1, &seq2)) break; - if (!sync.hidden) { + if (!sync_rec.hidden) { seq_range_array_add_range(&ctx->flag_updates, seq1, seq2); } else if (array_is_created(&ctx->modseq_updates)) { @@ -190,14 +199,14 @@ return &ctx->ctx; } -static int +static bool index_mailbox_sync_next_expunge(struct index_mailbox_sync_context *ctx, struct mailbox_sync_rec *sync_rec_r) { const struct seq_range *range; if (ctx->expunge_pos == 0) - return 0; + return FALSE; /* expunges is a sorted array of sequences. it's easiest for us to print them from end to beginning. */ @@ -211,7 +220,7 @@ sync_rec_r->seq1 = range->seq1; sync_rec_r->seq2 = range->seq2; sync_rec_r->type = MAILBOX_SYNC_TYPE_EXPUNGE; - return 1; + return TRUE; } bool index_mailbox_sync_next(struct mailbox_sync_context *_ctx, @@ -231,7 +240,7 @@ sync_rec_r->seq1 = range[ctx->flag_update_idx].seq1; sync_rec_r->seq2 = range[ctx->flag_update_idx].seq2; ctx->flag_update_idx++; - return 1; + return TRUE; } if (array_is_created(&ctx->modseq_updates)) { range = array_get(&ctx->modseq_updates, &count); @@ -240,7 +249,7 @@ sync_rec_r->seq1 = range[ctx->modseq_update_idx].seq1; sync_rec_r->seq2 = range[ctx->modseq_update_idx].seq2; ctx->modseq_update_idx++; - return 1; + return TRUE; } }
--- a/src/lib-storage/index/maildir/maildir-copy.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-copy.c Mon Jun 09 05:11:18 2008 +0300 @@ -106,15 +106,24 @@ { struct maildir_mailbox *dest_mbox = (struct maildir_mailbox *)t->ictx.ibox; - struct maildir_mailbox *src_mbox = - (struct maildir_mailbox *)mail->box; + struct maildir_mailbox *src_mbox; struct maildir_save_context *ctx; struct hardlink_ctx do_ctx; - const char *filename = NULL; + const char *path, *filename = NULL; uint32_t seq; i_assert((t->ictx.flags & MAILBOX_TRANSACTION_FLAG_EXTERNAL) != 0); + if (strcmp(mail->box->storage->name, MAILDIR_STORAGE_NAME) == 0) + src_mbox = (struct maildir_mailbox *)mail->box; + else if (strcmp(mail->box->storage->name, "raw") == 0) { + /* deliver uses raw format */ + src_mbox = NULL; + } else { + /* Can't hard link files from the source storage */ + return 0; + } + if (t->save_ctx == NULL) t->save_ctx = maildir_save_transaction_init(t); ctx = t->save_ctx; @@ -127,7 +136,7 @@ memset(&do_ctx, 0, sizeof(do_ctx)); do_ctx.dest_path = str_new(default_pool, 512); - if (dest_mbox->storage->copy_preserve_filename) { + if (dest_mbox->storage->copy_preserve_filename && src_mbox != NULL) { enum maildir_uidlist_rec_flag src_flags; const char *src_fname; @@ -175,15 +184,26 @@ } } else #endif -{ + { /* keywords, hardlink to tmp/ with basename and later when we have uidlist locked, move it to new/cur. */ str_printfa(do_ctx.dest_path, "%s/tmp/%s", dest_mbox->path, do_ctx.dest_fname); do_ctx.base_end_pos = str_len(do_ctx.dest_path); } - if (maildir_file_do(src_mbox, mail->uid, do_hardlink, &do_ctx) < 0) - return -1; + if (src_mbox != NULL) { + /* maildir */ + if (maildir_file_do(src_mbox, mail->uid, + do_hardlink, &do_ctx) < 0) + return -1; + } else { + /* raw / deliver */ + if (mail_get_special(mail, MAIL_FETCH_UIDL_FILE_NAME, + &path) < 0 || *path == '\0') + return 0; + if (do_hardlink(dest_mbox, path, &do_ctx) < 0) + return -1; + } if (!do_ctx.success) { /* couldn't copy with hardlinking, fallback to copying */ @@ -223,7 +243,6 @@ int ret; if (mbox->storage->copy_with_hardlinks && - mail->box->storage == mbox->ibox.box.storage && maildir_compatible_file_modes(&mbox->ibox.box, mail->box)) { T_BEGIN { ret = maildir_copy_hardlink(t, mail, flags,
--- a/src/lib-storage/index/maildir/maildir-mail.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-mail.c Mon Jun 09 05:11:18 2008 +0300 @@ -6,6 +6,7 @@ #include "maildir-storage.h" #include "maildir-filename.h" #include "maildir-uidlist.h" +#include "maildir-sync.h" #include <stdlib.h> #include <fcntl.h> @@ -143,13 +144,32 @@ const char **fname_r) { enum maildir_uidlist_rec_flag flags; + struct mail_index_view *view; + uint32_t seq; + bool exists; *fname_r = maildir_uidlist_lookup(mbox->uidlist, mail->uid, &flags); - if (*fname_r == NULL) { - mail_set_expunged(mail); - return FALSE; + if (*fname_r != NULL) + return TRUE; + + /* file exists in index file, but not in dovecot-uidlist anymore. */ + mail_set_expunged(mail); + + /* one reason this could happen is if we delayed opening + dovecot-uidlist and we're trying to open a mail that got recently + expunged. Let's test this theory first: */ + (void)mail_index_refresh(mbox->ibox.index); + view = mail_index_view_open(mbox->ibox.index); + exists = mail_index_lookup_seq(view, mail->uid, &seq); + mail_index_view_close(&view); + + if (exists) { + /* the message still exists in index. this means there's some + kind of a desync, which doesn't get fixed if cur/ mtime is + the same as in index. fix this by forcing a resync. */ + (void)maildir_storage_sync_force(mbox, mail->uid); } - return TRUE; + return FALSE; } static int maildir_get_pop3_state(struct index_mail *mail) @@ -274,14 +294,16 @@ do some extra checks here to catch potential cache bugs. */ if (vsize && mail->data.virtual_size != size) { mail_cache_set_corrupted(mail->ibox->cache, - "Corrupted virtual size: " + "Corrupted virtual size for uid=%u: " "%"PRIuUOFF_T" != %"PRIuUOFF_T, + mail->mail.mail.uid, mail->data.virtual_size, size); mail->data.virtual_size = size; } else if (!vsize && mail->data.physical_size != size) { mail_cache_set_corrupted(mail->ibox->cache, - "Corrupted physical size: " + "Corrupted physical size for uid=%u: " "%"PRIuUOFF_T" != %"PRIuUOFF_T, + mail->mail.mail.uid, mail->data.physical_size, size); mail->data.physical_size = size; } @@ -390,7 +412,7 @@ { struct index_mail *mail = (struct index_mail *)_mail; struct maildir_mailbox *mbox = (struct maildir_mailbox *)mail->ibox; - const char *path, *fname, *end; + const char *path, *fname, *end, *uidl; if (field == MAIL_FETCH_UIDL_FILE_NAME) { if (_mail->uid != 0) { @@ -405,6 +427,11 @@ end = strchr(fname, MAILDIR_INFO_SEP); *value_r = end == NULL ? fname : t_strdup_until(fname, end); return 0; + } else if (field == MAIL_FETCH_UIDL_BACKEND) { + uidl = maildir_uidlist_lookup_ext(mbox->uidlist, _mail->uid, + MAILDIR_UIDLIST_REC_EXT_POP3_UIDL); + *value_r = uidl != NULL ? uidl : ""; + return 0; } return index_mail_get_special(_mail, field, value_r); @@ -432,6 +459,34 @@ return index_mail_init_stream(mail, hdr_size, body_size, stream_r); } +static void maildir_mail_set_cache_corrupted(struct mail *_mail, + enum mail_fetch_field field) +{ + struct index_mail *mail = (struct index_mail *)_mail; + struct maildir_mailbox *mbox = (struct maildir_mailbox *)mail->ibox; + enum maildir_uidlist_rec_flag flags; + const char *fname; + uoff_t size; + + if (field == MAIL_FETCH_VIRTUAL_SIZE) { + /* make sure it gets removed from uidlist. + if it's in file name, we can't really do more than log it. */ + fname = maildir_uidlist_lookup(mbox->uidlist, + _mail->uid, &flags); + if (maildir_filename_get_size(fname, MAILDIR_EXTRA_VIRTUAL_SIZE, + &size)) { + i_error("Maildir filename has wrong W value: %s/%s", + mbox->path, fname); + } else if (maildir_uidlist_lookup_ext(mbox->uidlist, _mail->uid, + MAILDIR_UIDLIST_REC_EXT_VSIZE) != NULL) { + maildir_uidlist_set_ext(mbox->uidlist, _mail->uid, + MAILDIR_UIDLIST_REC_EXT_VSIZE, + NULL); + } + } + index_mail_set_cache_corrupted(_mail, field); +} + struct mail_vfuncs maildir_mail_vfuncs = { index_mail_close, index_mail_free, @@ -456,6 +511,6 @@ index_mail_update_flags, index_mail_update_keywords, index_mail_expunge, - index_mail_set_cache_corrupted, + maildir_mail_set_cache_corrupted, index_mail_get_index_mail };
--- a/src/lib-storage/index/maildir/maildir-save.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-save.c Mon Jun 09 05:11:18 2008 +0300 @@ -221,11 +221,11 @@ MAILDIR_EXTRA_FILE_SIZE, mf->size); } - /*if (mf->vsize != (uoff_t)-1) { + if (mf->vsize != (uoff_t)-1) { basename = t_strdup_printf("%s,%c=%"PRIuUOFF_T, basename, MAILDIR_EXTRA_VIRTUAL_SIZE, mf->vsize); - }*/ + } if (mf->keywords_count == 0) { if ((mf->flags & MAIL_FLAGS_MASK) == MAIL_RECENT) { @@ -593,81 +593,99 @@ return 0; } +static int +maildir_transaction_save_commit_pre_sync(struct maildir_save_context *ctx) +{ + struct maildir_transaction_context *t = + (struct maildir_transaction_context *)ctx->ctx.transaction; + struct maildir_mailbox *mbox = ctx->mbox; + uint32_t uid, first_uid, next_uid; + int ret; + + /* we'll need to keep the lock past the sync deinit */ + ret = maildir_uidlist_lock(mbox->uidlist); + i_assert(ret > 0); + + if (maildir_sync_index_begin(mbox, NULL, &ctx->sync_ctx) < 0) + return -1; + + if (maildir_sync_header_refresh(mbox) < 0) + return -1; + if (maildir_uidlist_refresh_fast_init(mbox->uidlist) < 0) + return 1; + + ctx->keywords_sync_ctx = + maildir_sync_get_keywords_sync_ctx(ctx->sync_ctx); + + /* now that uidlist is locked, make sure all the existing mails + have been added to index. we don't really look into the + maildir, just add all the new mails listed in + dovecot-uidlist to index. */ + if (maildir_sync_index(ctx->sync_ctx, TRUE) < 0) + return -1; + + /* if messages were added to index, assign them UIDs */ + first_uid = maildir_uidlist_get_next_uid(mbox->uidlist); + i_assert(first_uid != 0); + mail_index_append_assign_uids(ctx->trans, first_uid, &next_uid); + i_assert(next_uid = first_uid + ctx->files_count); + + /* these mails are all recent in our session */ + for (uid = first_uid; uid < next_uid; uid++) + index_mailbox_set_recent_uid(&mbox->ibox, uid); + + if (!mbox->ibox.keep_recent) { + /* maildir_sync_index() dropped recent flags from + existing messages. we'll still need to drop recent + flags from these newly added messages. */ + mail_index_update_header(ctx->trans, + offsetof(struct mail_index_header, + first_recent_uid), + &next_uid, sizeof(next_uid), FALSE); + } + + /* this will work even if index isn't updated */ + *t->ictx.first_saved_uid = first_uid; + *t->ictx.last_saved_uid = next_uid - 1; + return 0; +} + int maildir_transaction_save_commit_pre(struct maildir_save_context *ctx) { struct maildir_transaction_context *t = (struct maildir_transaction_context *)ctx->ctx.transaction; struct maildir_filename *mf; - uint32_t seq, uid, first_uid, next_uid; + uint32_t seq; enum maildir_uidlist_rec_flag flags; - bool newdir, new_changed, cur_changed, sync_commit = FALSE; + enum maildir_uidlist_sync_flags sync_flags; + bool newdir, new_changed, cur_changed; int ret; i_assert(ctx->output == NULL); i_assert(ctx->finished); + sync_flags = MAILDIR_UIDLIST_SYNC_PARTIAL | + MAILDIR_UIDLIST_SYNC_NOREFRESH; + /* if we want to assign UIDs or keywords, we require uidlist lock */ if ((t->ictx.flags & MAILBOX_TRANSACTION_FLAG_ASSIGN_UIDS) == 0 && !ctx->have_keywords) { /* assign the UIDs if we happen to get a lock */ - ctx->locked = maildir_uidlist_try_lock(ctx->mbox->uidlist) > 0; - } else { - if (maildir_uidlist_lock(ctx->mbox->uidlist) <= 0) { - /* error or timeout - our transaction is broken */ - maildir_transaction_save_rollback(ctx); - return -1; - } - ctx->locked = TRUE; + sync_flags |= MAILDIR_UIDLIST_SYNC_TRYLOCK; + } + ret = maildir_uidlist_sync_init(ctx->mbox->uidlist, sync_flags, + &ctx->uidlist_sync_ctx); + if (ret < 0) { + maildir_transaction_save_rollback(ctx); + return -1; } + ctx->locked = ret > 0; if (ctx->locked) { - ret = maildir_uidlist_sync_init(ctx->mbox->uidlist, - MAILDIR_UIDLIST_SYNC_PARTIAL, - &ctx->uidlist_sync_ctx); - i_assert(ret > 0); /* already locked, shouldn't fail */ - - if (maildir_sync_index_begin(ctx->mbox, NULL, - &ctx->sync_ctx) < 0) { + if (maildir_transaction_save_commit_pre_sync(ctx) < 0) { maildir_transaction_save_rollback(ctx); return -1; } - - ctx->keywords_sync_ctx = - maildir_sync_get_keywords_sync_ctx(ctx->sync_ctx); - - /* now that uidlist is locked, make sure all the existing mails - have been added to index. we don't really look into the - maildir, just add all the new mails listed in - dovecot-uidlist to index. */ - if (maildir_sync_index(ctx->sync_ctx, TRUE) < 0) { - maildir_transaction_save_rollback(ctx); - return -1; - } - sync_commit = TRUE; - - /* if messages were added to index, assign them UIDs */ - first_uid = maildir_uidlist_get_next_uid(ctx->mbox->uidlist); - i_assert(first_uid != 0); - mail_index_append_assign_uids(ctx->trans, first_uid, &next_uid); - i_assert(next_uid = first_uid + ctx->files_count); - - /* these mails are all recent in our session */ - for (uid = first_uid; uid < next_uid; uid++) - index_mailbox_set_recent_uid(&ctx->mbox->ibox, uid); - - if (!ctx->mbox->ibox.keep_recent) { - /* maildir_sync_index() dropped recent flags from - existing messages. we'll still need to drop recent - flags from these newly added messages. */ - mail_index_update_header(ctx->trans, - offsetof(struct mail_index_header, - first_recent_uid), - &next_uid, sizeof(next_uid), FALSE); - } - - /* this will work even if index isn't updated */ - *t->ictx.first_saved_uid = first_uid; - *t->ictx.last_saved_uid = next_uid - 1; } else { /* since we couldn't lock uidlist, we'll have to drop the appends to index. */ @@ -746,11 +764,11 @@ mail_free(&ctx->mail); } - if (sync_commit) { + if (ctx->locked) { /* It doesn't matter if index syncing fails */ ctx->keywords_sync_ctx = NULL; (void)maildir_sync_index_finish(&ctx->sync_ctx, - ret < 0, !sync_commit); + ret < 0, FALSE); } if (ret < 0) {
--- a/src/lib-storage/index/maildir/maildir-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -48,14 +48,15 @@ maildir_list_rename_mailbox(struct mailbox_list *list, const char *oldname, const char *newname); static int -maildir_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx - ATTR_UNUSED, +maildir_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name, enum mailbox_list_file_type type, enum mailbox_info_flags *flags_r); static int maildirplusplus_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name, enum mailbox_list_file_type type, enum mailbox_info_flags *flags_r); @@ -864,6 +865,7 @@ maildir_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx ATTR_UNUSED, const char *dir, const char *fname, + const char *mailbox_name ATTR_UNUSED, enum mailbox_list_file_type type, enum mailbox_info_flags *flags_r) { @@ -913,6 +915,7 @@ static int maildirplusplus_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name ATTR_UNUSED, enum mailbox_list_file_type type, enum mailbox_info_flags *flags_r) {
--- a/src/lib-storage/index/maildir/maildir-storage.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-storage.h Mon Jun 09 05:11:18 2008 +0300 @@ -57,6 +57,7 @@ struct maildir_index_header { uint32_t new_check_time, new_mtime, new_mtime_nsecs; uint32_t cur_check_time, cur_mtime, cur_mtime_nsecs; + uint32_t uidlist_mtime, uidlist_mtime_nsecs, uidlist_size; }; struct maildir_list_index_record {
--- a/src/lib-storage/index/maildir/maildir-sync-index.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-sync-index.c Mon Jun 09 05:11:18 2008 +0300 @@ -186,6 +186,46 @@ return 0; } +static bool +maildir_index_header_has_changed(const struct maildir_index_header *old_hdr, + const struct maildir_index_header *new_hdr) +{ +#define DIR_DELAYED_REFRESH(hdr, name) \ + ((hdr)->name ## _check_time <= \ + (hdr)->name ## _mtime + MAILDIR_SYNC_SECS) + + if (old_hdr->new_mtime != new_hdr->new_mtime || + old_hdr->new_mtime_nsecs != new_hdr->new_mtime_nsecs || + old_hdr->cur_mtime != new_hdr->cur_mtime || + old_hdr->cur_mtime_nsecs != new_hdr->cur_mtime_nsecs || + old_hdr->uidlist_mtime != new_hdr->uidlist_mtime || + old_hdr->uidlist_mtime_nsecs != new_hdr->uidlist_mtime_nsecs || + old_hdr->uidlist_size != new_hdr->uidlist_size) + return TRUE; + + return DIR_DELAYED_REFRESH(old_hdr, new) != + DIR_DELAYED_REFRESH(new_hdr, new) || + DIR_DELAYED_REFRESH(old_hdr, cur) != + DIR_DELAYED_REFRESH(new_hdr, cur); +} + +static void +maildir_sync_index_update_ext_header(struct maildir_index_sync_context *ctx) +{ + struct maildir_mailbox *mbox = ctx->mbox; + const void *data; + size_t data_size; + + mail_index_get_header_ext(mbox->ibox.view, mbox->maildir_ext_id, + &data, &data_size); + if (data_size != sizeof(mbox->maildir_hdr) || + maildir_index_header_has_changed(data, &mbox->maildir_hdr)) { + mail_index_update_header_ext(ctx->trans, mbox->maildir_ext_id, + 0, &mbox->maildir_hdr, + sizeof(mbox->maildir_hdr)); + } +} + int maildir_sync_index_finish(struct maildir_index_sync_context **_ctx, bool failed, bool cancel) { @@ -198,6 +238,8 @@ if (ret < 0 || cancel) mail_index_sync_rollback(&ctx->sync_ctx); else { + maildir_sync_index_update_ext_header(ctx); + /* Set syncing_commit=TRUE so that if any sync callbacks try to access mails which got lost (eg. expunge callback trying to open the file which was just unlinked) we don't try to @@ -219,43 +261,6 @@ return ret; } -static bool -maildir_index_header_has_changed(const struct maildir_index_header *old_hdr, - const struct maildir_index_header *new_hdr) -{ -#define DIR_DELAYED_REFRESH(hdr, name) \ - ((hdr)->name ## _check_time <= \ - (hdr)->name ## _mtime + MAILDIR_SYNC_SECS) - - if (old_hdr->new_mtime != new_hdr->new_mtime || - old_hdr->cur_mtime != new_hdr->cur_mtime || - old_hdr->new_mtime_nsecs != new_hdr->new_mtime_nsecs || - old_hdr->cur_mtime_nsecs != new_hdr->cur_mtime_nsecs) - return TRUE; - - return DIR_DELAYED_REFRESH(old_hdr, new) != - DIR_DELAYED_REFRESH(new_hdr, new) || - DIR_DELAYED_REFRESH(old_hdr, cur) != - DIR_DELAYED_REFRESH(new_hdr, cur); -} - -static void -maildir_index_update_ext_header(struct maildir_mailbox *mbox, - struct mail_index_transaction *trans) -{ - const void *data; - size_t data_size; - - mail_index_get_header_ext(mbox->ibox.view, mbox->maildir_ext_id, - &data, &data_size); - if (data_size != sizeof(mbox->maildir_hdr) || - maildir_index_header_has_changed(data, &mbox->maildir_hdr)) { - mail_index_update_header_ext(trans, mbox->maildir_ext_id, 0, - &mbox->maildir_hdr, - sizeof(mbox->maildir_hdr)); - } -} - int maildir_sync_index(struct maildir_index_sync_context *ctx, bool partial) { @@ -272,6 +277,7 @@ const char *filename; ARRAY_TYPE(keyword_indexes) idx_keywords; uint32_t uid_validity, next_uid, hdr_next_uid, first_recent_uid; + uint32_t first_uid; unsigned int changes = 0; int ret = 0; bool expunged, full_rescan = FALSE; @@ -279,6 +285,7 @@ i_assert(!mbox->syncing_commit); i_assert(maildir_uidlist_is_locked(mbox->uidlist)); + first_uid = 1; hdr = mail_index_get_header(view); uid_validity = maildir_uidlist_get_uid_validity(mbox->uidlist); if (uid_validity != hdr->uid_validity && @@ -289,8 +296,10 @@ i_warning("Maildir %s: UIDVALIDITY changed (%u -> %u)", mbox->path, hdr->uid_validity, uid_validity); mail_index_reset(trans); + index_mailbox_reset_uidvalidity(&mbox->ibox); maildir_uidlist_set_next_uid(mbox->uidlist, 1, TRUE); + first_uid = hdr->messages_count + 1; memset(&empty_hdr, 0, sizeof(empty_hdr)); empty_hdr.next_uid = 1; hdr = &empty_hdr; @@ -438,8 +447,13 @@ appended messages. */ view2 = mail_index_transaction_open_updated_view(trans); if (mail_index_lookup_seq_range(view2, first_recent_uid, (uint32_t)-1, - &seq, &seq2)) + &seq, &seq2) && seq2 >= first_uid) { + if (seq < first_uid) { + /* UIDVALIDITY changed, skip over the old messages */ + seq = first_uid; + } index_mailbox_set_recent_seq(&mbox->ibox, view2, seq, seq2); + } mail_index_view_close(&view2); if (ctx->uidlist_sync_ctx != NULL) { @@ -452,7 +466,6 @@ if (ctx->changed) mbox->maildir_hdr.cur_mtime = time(NULL); - maildir_index_update_ext_header(mbox, trans); if (uid_validity == 0) { uid_validity = hdr->uid_validity != 0 ?
--- a/src/lib-storage/index/maildir/maildir-sync.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-sync.c Mon Jun 09 05:11:18 2008 +0300 @@ -403,19 +403,11 @@ if (new_dir) { ctx->mbox->maildir_hdr.new_check_time = now; ctx->mbox->maildir_hdr.new_mtime = st.st_mtime; -#ifdef HAVE_STAT_TV_NSEC - ctx->mbox->maildir_hdr.new_mtime_nsecs = st.st_mtim.tv_nsec; -#else - ctx->mbox->maildir_hdr.new_mtime_nsecs = 0; -#endif + ctx->mbox->maildir_hdr.new_mtime_nsecs = ST_MTIME_NSEC(st); } else { ctx->mbox->maildir_hdr.cur_check_time = now; ctx->mbox->maildir_hdr.cur_mtime = st.st_mtime; -#ifdef HAVE_STAT_TV_NSEC - ctx->mbox->maildir_hdr.cur_mtime_nsecs = st.st_mtim.tv_nsec; -#else - ctx->mbox->maildir_hdr.cur_mtime_nsecs = 0; -#endif + ctx->mbox->maildir_hdr.cur_mtime_nsecs = ST_MTIME_NSEC(st); } src = t_str_new(1024); @@ -512,7 +504,7 @@ (move_count <= MAILDIR_RENAME_RESCAN_COUNT ? 0 : 1); } -static int maildir_header_refresh(struct maildir_mailbox *mbox) +int maildir_sync_header_refresh(struct maildir_mailbox *mbox) { const void *data; size_t data_size; @@ -529,10 +521,8 @@ return 0; } - if (data_size != sizeof(mbox->maildir_hdr)) - i_warning("Maildir %s: Invalid header record size", mbox->path); - else - memcpy(&mbox->maildir_hdr, data, sizeof(mbox->maildir_hdr)); + memcpy(&mbox->maildir_hdr, data, + I_MIN(sizeof(mbox->maildir_hdr), data_size)); return 0; } @@ -540,13 +530,6 @@ const char *new_dir, const char *cur_dir, bool *new_changed_r, bool *cur_changed_r) { -#ifdef HAVE_STAT_TV_NSEC -# define DIR_NSECS_CHANGED(st, hdr, name) \ - ((unsigned int)(st).st_mtim.tv_nsec != (hdr)->name ## _mtime_nsecs) -#else -# define DIR_NSECS_CHANGED(st, hdr, name) 0 -#endif - #define DIR_DELAYED_REFRESH(hdr, name) \ ((hdr)->name ## _check_time <= \ (hdr)->name ## _mtime + MAILDIR_SYNC_SECS && \ @@ -555,14 +538,14 @@ #define DIR_MTIME_CHANGED(st, hdr, name) \ ((st).st_mtime != (time_t)(hdr)->name ## _mtime || \ - DIR_NSECS_CHANGED(st, hdr, name)) + !ST_NTIMES_EQUAL(ST_MTIME_NSEC(st), (hdr)->name ## _mtime_nsecs)) struct maildir_index_header *hdr = &mbox->maildir_hdr; struct stat new_st, cur_st; bool refreshed = FALSE, check_new = FALSE, check_cur = FALSE; if (mbox->maildir_hdr.new_mtime == 0) { - if (maildir_header_refresh(mbox) < 0) + if (maildir_sync_header_refresh(mbox) < 0) return -1; if (mbox->maildir_hdr.new_mtime == 0) { /* first sync */ @@ -577,7 +560,7 @@ if (DIR_DELAYED_REFRESH(hdr, new) || DIR_DELAYED_REFRESH(hdr, cur)) { /* refresh index and try again */ - if (maildir_header_refresh(mbox) < 0) + if (maildir_sync_header_refresh(mbox) < 0) return -1; refreshed = TRUE; @@ -610,7 +593,7 @@ break; /* refresh index and try again */ - if (maildir_header_refresh(mbox) < 0) + if (maildir_sync_header_refresh(mbox) < 0) return -1; refreshed = TRUE; } @@ -760,7 +743,6 @@ &ctx->uidlist_sync_ctx); if (ret <= 0) { /* failure / timeout */ - i_assert(ret < 0 || !forced); return ret; } ctx->locked = maildir_uidlist_is_locked(ctx->mbox->uidlist);
--- a/src/lib-storage/index/maildir/maildir-sync.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-sync.h Mon Jun 09 05:11:18 2008 +0300 @@ -24,6 +24,8 @@ maildir_storage_sync_init(struct mailbox *box, enum mailbox_sync_flags flags); int maildir_storage_sync_force(struct maildir_mailbox *mbox, uint32_t uid); +int maildir_sync_header_refresh(struct maildir_mailbox *mbox); + int maildir_sync_index_begin(struct maildir_mailbox *mbox, struct maildir_sync_context *maildir_sync_ctx, struct maildir_index_sync_context **ctx_r);
--- a/src/lib-storage/index/maildir/maildir-uidlist.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-uidlist.c Mon Jun 09 05:11:18 2008 +0300 @@ -49,6 +49,7 @@ /* how many seconds to wait before overriding uidlist.lock */ #define UIDLIST_LOCK_STALE_TIMEOUT (60*2) +#define UIDLIST_VERSION 3 #define UIDLIST_COMPRESS_PERCENTAGE 75 #define UIDLIST_IS_LOCKED(uidlist) \ @@ -90,6 +91,7 @@ unsigned int recreate:1; unsigned int initial_read:1; + unsigned int initial_hdr_read:1; unsigned int initial_sync:1; }; @@ -124,7 +126,7 @@ struct maildir_uidlist_rec **rec_r); static int maildir_uidlist_lock_timeout(struct maildir_uidlist *uidlist, - bool nonblock) + bool nonblock, bool refresh) { struct mailbox *box = &uidlist->ibox->box; const char *control_dir, *path; @@ -170,22 +172,25 @@ uidlist->lock_count++; - /* make sure we have the latest changes before changing anything */ - if (maildir_uidlist_refresh(uidlist) < 0) { - maildir_uidlist_unlock(uidlist); - return -1; + if (refresh) { + /* make sure we have the latest changes before + changing anything */ + if (maildir_uidlist_refresh(uidlist) < 0) { + maildir_uidlist_unlock(uidlist); + return -1; + } } return 1; } int maildir_uidlist_lock(struct maildir_uidlist *uidlist) { - return maildir_uidlist_lock_timeout(uidlist, FALSE); + return maildir_uidlist_lock_timeout(uidlist, FALSE, TRUE); } int maildir_uidlist_try_lock(struct maildir_uidlist *uidlist) { - return maildir_uidlist_lock_timeout(uidlist, TRUE); + return maildir_uidlist_lock_timeout(uidlist, TRUE, TRUE); } int maildir_uidlist_lock_touch(struct maildir_uidlist *uidlist) @@ -293,6 +298,36 @@ (*rec1)->uid > (*rec2)->uid ? 1 : 0; } +static void ATTR_FORMAT(2, 3) +maildir_uidlist_set_corrupted(struct maildir_uidlist *uidlist, + const char *fmt, ...) +{ + struct mail_storage *storage = uidlist->ibox->box.storage; + va_list args; + + va_start(args, fmt); + mail_storage_set_critical(storage, "Broken file %s line %u: %s", + uidlist->path, uidlist->read_line_count, + t_strdup_vprintf(fmt, args)); + va_end(args); +} + +static void maildir_uidlist_update_hdr(struct maildir_uidlist *uidlist, + const struct stat *st) +{ + struct maildir_index_header *mhdr; + + if (uidlist->mbox == NULL) { + /* dbox is using this */ + return; + } + + mhdr = &uidlist->mbox->maildir_hdr; + mhdr->uidlist_mtime = st->st_mtime; + mhdr->uidlist_mtime_nsecs = ST_MTIME_NSEC(*st); + mhdr->uidlist_size = st->st_size; +} + static void maildir_uidlist_records_array_delete(struct maildir_uidlist *uidlist, struct maildir_uidlist_rec *rec) @@ -341,22 +376,8 @@ return TRUE; } -static void ATTR_FORMAT(2, 3) -maildir_uidlist_set_corrupted(struct maildir_uidlist *uidlist, - const char *fmt, ...) -{ - struct mail_storage *storage = uidlist->ibox->box.storage; - va_list args; - - va_start(args, fmt); - mail_storage_set_critical(storage, "Broken file %s line %u: %s", - uidlist->path, uidlist->read_line_count, - t_strdup_vprintf(fmt, args)); - va_end(args); -} - -static int maildir_uidlist_next(struct maildir_uidlist *uidlist, - const char *line) +static bool maildir_uidlist_next(struct maildir_uidlist *uidlist, + const char *line) { struct maildir_uidlist_rec *rec, *old_rec; uint32_t uid; @@ -371,19 +392,19 @@ /* invalid file */ maildir_uidlist_set_corrupted(uidlist, "Invalid data: %s", line); - return 0; + return FALSE; } if (uid <= uidlist->prev_read_uid) { maildir_uidlist_set_corrupted(uidlist, "UIDs not ordered (%u > %u)", uid, uidlist->prev_read_uid); - return 0; + return FALSE; } uidlist->prev_read_uid = uid; if (uid <= uidlist->last_seen_uid) { /* we already have this */ - return 1; + return TRUE; } uidlist->last_seen_uid = uid; @@ -391,7 +412,7 @@ maildir_uidlist_set_corrupted(uidlist, "UID larger than next_uid (%u >= %u)", uid, uidlist->next_uid); - return 0; + return FALSE; } rec = p_new(uidlist->record_pool, struct maildir_uidlist_rec, 1); @@ -400,7 +421,7 @@ while (*line == ' ') line++; - if (uidlist->version == 3) { + if (uidlist->version == UIDLIST_VERSION) { /* read extended fields */ bool ret; @@ -411,16 +432,25 @@ if (!ret) { maildir_uidlist_set_corrupted(uidlist, "Invalid extended fields: %s", line); - return 0; + return FALSE; } } + if (strchr(line, '/') != NULL) { + maildir_uidlist_set_corrupted(uidlist, + "%s: Broken filename at line %u: %s", + uidlist->path, uidlist->read_line_count, line); + return FALSE; + } + old_rec = hash_lookup(uidlist->files, line); if (old_rec != NULL) { /* This can happen if expunged file is moved back and the file was appended to uidlist. */ - i_warning("%s: Duplicate file entry at line %u: %s", - uidlist->path, uidlist->read_line_count, line); + i_warning("%s: Duplicate file entry at line %u: " + "%s (uid %u -> %u)", + uidlist->path, uidlist->read_line_count, line, + old_rec->uid, uid); /* Delete the old UID */ maildir_uidlist_records_array_delete(uidlist, old_rec); /* Replace the old record with this new one */ @@ -432,7 +462,7 @@ rec->filename = p_strdup(uidlist->record_pool, line); hash_insert(uidlist->files, rec->filename, rec); array_append(&uidlist->records, &rec, 1); - return 1; + return TRUE; } static int maildir_uidlist_read_header(struct maildir_uidlist *uidlist, @@ -474,7 +504,7 @@ return 0; } break; - case 3: + case UIDLIST_VERSION: ext_hdr = uidlist->hdr_extensions; str_truncate(ext_hdr, 0); while (*line != '\0') T_BEGIN { @@ -623,6 +653,7 @@ uidlist->fd_ino = st.st_ino; uidlist->fd_size = st.st_size; uidlist->last_read_offset = input->v_offset; + maildir_uidlist_update_hdr(uidlist, &st); } else { /* I/O error */ if (input->stream_errno == ESTALE && try_retry) @@ -643,18 +674,15 @@ } static int -maildir_uidlist_has_changed(struct maildir_uidlist *uidlist, bool *recreated_r) +maildir_uidlist_stat(struct maildir_uidlist *uidlist, struct stat *st_r) { struct mail_storage *storage = uidlist->ibox->box.storage; - struct stat st; - - *recreated_r = FALSE; if ((storage->flags & MAIL_STORAGE_FLAG_NFS_FLUSH_STORAGE) != 0) { nfs_flush_file_handle_cache(uidlist->path); nfs_flush_attr_cache_unlocked(uidlist->path); } - if (nfs_safe_stat(uidlist->path, &st) < 0) { + if (nfs_safe_stat(uidlist->path, st_r) < 0) { if (errno != ENOENT) { mail_storage_set_critical(storage, "stat(%s) failed: %m", uidlist->path); @@ -662,6 +690,20 @@ } return 0; } + return 1; +} + +static int +maildir_uidlist_has_changed(struct maildir_uidlist *uidlist, bool *recreated_r) +{ + struct mail_storage *storage = uidlist->ibox->box.storage; + struct stat st; + int ret; + + *recreated_r = FALSE; + + if ((ret = maildir_uidlist_stat(uidlist, &st)) <= 0) + return ret; if (st.st_ino != uidlist->fd_ino || !CMP_DEV_T(st.st_dev, uidlist->fd_dev)) { @@ -717,11 +759,40 @@ /* ESTALE - try reopening and rereading */ maildir_uidlist_close(uidlist); } - if (ret >= 0) + if (ret >= 0) { uidlist->initial_read = TRUE; + uidlist->initial_hdr_read = TRUE; + } return ret; } +int maildir_uidlist_refresh_fast_init(struct maildir_uidlist *uidlist) +{ + const struct maildir_index_header *mhdr = &uidlist->mbox->maildir_hdr; + const struct mail_index_header *hdr; + struct stat st; + int ret; + + if (uidlist->fd != -1) + return maildir_uidlist_refresh(uidlist); + + if ((ret = maildir_uidlist_stat(uidlist, &st)) < 0) + return ret; + + if (st.st_size == mhdr->uidlist_size && + st.st_mtime == mhdr->uidlist_mtime && + ST_NTIMES_EQUAL(ST_MTIME_NSEC(st), mhdr->uidlist_mtime_nsecs)) { + /* index is up-to-date */ + hdr = mail_index_get_header(uidlist->mbox->ibox.view); + uidlist->uid_validity = hdr->uid_validity; + uidlist->next_uid = hdr->next_uid; + uidlist->initial_hdr_read = TRUE; + return 1; + } else { + return maildir_uidlist_refresh(uidlist); + } +} + static struct maildir_uidlist_rec * maildir_uidlist_lookup_rec(struct maildir_uidlist *uidlist, uint32_t uid, unsigned int *idx_r) @@ -763,12 +834,16 @@ fname = maildir_uidlist_lookup_nosync(uidlist, uid, flags_r); if (fname == NULL) { - if (uidlist->fd != -1 || uidlist->mbox == NULL) - return NULL; - - /* the uidlist doesn't exist. */ - if (maildir_storage_sync_force(uidlist->mbox, uid) < 0) - return NULL; + if (uidlist->fd != -1 || uidlist->mbox == NULL) { + /* refresh uidlist and check again in case it was added + after the last mailbox sync */ + if (maildir_uidlist_refresh(uidlist) < 0) + return NULL; + } else { + /* the uidlist doesn't exist. */ + if (maildir_storage_sync_force(uidlist->mbox, uid) < 0) + return NULL; + } /* try again */ fname = maildir_uidlist_lookup_nosync(uidlist, uid, flags_r); @@ -823,7 +898,7 @@ uint32_t maildir_uidlist_get_next_uid(struct maildir_uidlist *uidlist) { - return !uidlist->initial_read ? 0 : uidlist->next_uid; + return !uidlist->initial_hdr_read ? 0 : uidlist->next_uid; } void maildir_uidlist_set_uid_validity(struct maildir_uidlist *uidlist, @@ -877,8 +952,10 @@ p += len; } } - buffer_append_c(buf, key); - buffer_append(buf, value, strlen(value) + 1); + if (value != NULL) { + buffer_append_c(buf, key); + buffer_append(buf, value, strlen(value) + 1); + } buffer_append_c(buf, '\0'); rec->extensions = p_malloc(uidlist->record_pool, buf->used); @@ -917,7 +994,7 @@ if (output->offset == 0) { i_assert(first_idx == 0); - uidlist->version = 3; + uidlist->version = UIDLIST_VERSION; i_assert(uidlist->uid_validity != 0); i_assert(uidlist->next_uid > 0); @@ -960,7 +1037,6 @@ if (ret < 0) { mail_storage_set_critical(storage, "o_stream_send(%s) failed: %m", path); - (void)close(fd); return -1; } @@ -968,7 +1044,6 @@ if (fdatasync(fd) < 0) { mail_storage_set_critical(storage, "fdatasync(%s) failed: %m", path); - (void)close(fd); return -1; } } @@ -984,6 +1059,8 @@ uoff_t file_size; int i, fd, ret; + i_assert(uidlist->initial_read); + control_dir = mailbox_list_get_path(box->storage->list, box->name, MAILBOX_LIST_PATH_TYPE_CONTROL); temp_path = t_strconcat(control_dir, @@ -1031,14 +1108,14 @@ "unlink(%s) failed: %m", temp_path); } } else if (fstat(fd, &st) < 0) { - i_error("fstat(%s) failed: %m", temp_path); - (void)close(fd); + mail_storage_set_critical(box->storage, + "fstat(%s) failed: %m", temp_path); ret = -1; } else if (file_size != (uoff_t)st.st_size) { i_assert(!file_dotlock_is_locked(uidlist->dotlock)); - i_error("Maildir uidlist dotlock overridden: %s", + mail_storage_set_critical(box->storage, + "Maildir uidlist dotlock overridden: %s", uidlist->path); - (void)close(fd); ret = -1; } else { maildir_uidlist_close(uidlist); @@ -1048,7 +1125,10 @@ uidlist->fd_size = st.st_size; uidlist->last_read_offset = st.st_size; uidlist->recreate = FALSE; + maildir_uidlist_update_hdr(uidlist, &st); } + if (ret < 0) + (void)close(fd); return ret; } @@ -1066,9 +1146,29 @@ return ret; } +static bool maildir_uidlist_want_recreate(struct maildir_uidlist_sync_ctx *ctx) +{ + struct maildir_uidlist *uidlist = ctx->uidlist; + unsigned int min_rewrite_count; + + if (!uidlist->initial_read) + return FALSE; + + if (uidlist->recreate || uidlist->fd == -1 || + uidlist->version != UIDLIST_VERSION || + ctx->finish_change_counter != uidlist->change_counter) + return TRUE; + + min_rewrite_count = + (uidlist->read_records_count + ctx->new_files_count) * + UIDLIST_COMPRESS_PERCENTAGE / 100; + return min_rewrite_count >= array_count(&uidlist->records); +} + static int maildir_uidlist_sync_update(struct maildir_uidlist_sync_ctx *ctx) { struct maildir_uidlist *uidlist = ctx->uidlist; + struct stat st; uoff_t file_size; if (uidlist->uid_validity == 0) { @@ -1080,13 +1180,22 @@ hdr->uid_validity : (uint32_t)ioloop_time; } - if (ctx->uidlist->recreate || uidlist->fd == -1 || - uidlist->version != 3 || - ctx->finish_change_counter != ctx->uidlist->change_counter || - (uidlist->read_records_count + ctx->new_files_count) * - UIDLIST_COMPRESS_PERCENTAGE / 100 >= array_count(&uidlist->records)) + + if (maildir_uidlist_want_recreate(ctx)) return maildir_uidlist_recreate(uidlist); + if (uidlist->fd == -1) { + /* NOREFRESH flag used. we're just appending some messages. */ + i_assert(uidlist->initial_hdr_read); + + uidlist->fd = nfs_safe_open(uidlist->path, O_RDWR); + if (uidlist->fd == -1) { + mail_storage_set_critical(uidlist->ibox->box.storage, + "open(%s) failed: %m", uidlist->path); + return -1; + } + } + i_assert(ctx->first_unwritten_pos != (unsigned int)-1); if (lseek(uidlist->fd, 0, SEEK_END) < 0) { @@ -1099,7 +1208,19 @@ ctx->first_unwritten_pos, &file_size) < 0) return -1; - uidlist->last_read_offset = file_size; + if (fstat(uidlist->fd, &st) < 0) { + mail_storage_set_critical(uidlist->ibox->box.storage, + "fstat(%s) failed: %m", uidlist->path); + return -1; + } + if ((uoff_t)st.st_size != file_size) { + i_warning("%s: file size changed unexpectedly after write", + uidlist->path); + } else { + uidlist->fd_size = st.st_size; + uidlist->last_read_offset = st.st_size; + maildir_uidlist_update_hdr(uidlist, &st); + } return 0; } @@ -1124,27 +1245,26 @@ struct maildir_uidlist_sync_ctx **sync_ctx_r) { struct maildir_uidlist_sync_ctx *ctx; - bool locked; + bool nonblock, refresh, locked; int ret; - if ((sync_flags & MAILDIR_UIDLIST_SYNC_TRYLOCK) == 0) { - if ((ret = maildir_uidlist_lock(uidlist)) <= 0) + nonblock = (sync_flags & MAILDIR_UIDLIST_SYNC_TRYLOCK) != 0; + refresh = (sync_flags & MAILDIR_UIDLIST_SYNC_NOREFRESH) == 0; + + ret = maildir_uidlist_lock_timeout(uidlist, nonblock, refresh); + if (ret <= 0) { + if (ret < 0 || !nonblock) return ret; - } else { - if ((ret = maildir_uidlist_try_lock(uidlist)) < 0) - return -1; - if (ret == 0) { - /* couldn't lock it */ - if ((sync_flags & MAILDIR_UIDLIST_SYNC_FORCE) == 0) - return 0; - /* forcing the lock */ - } - } - locked = ret > 0; - if (!locked) { + /* couldn't lock it */ + if ((sync_flags & MAILDIR_UIDLIST_SYNC_FORCE) == 0) + return 0; + /* forcing the sync anyway */ if (maildir_uidlist_refresh(uidlist) < 0) return -1; + locked = FALSE; + } else { + locked = TRUE; } *sync_ctx_r = ctx = i_new(struct maildir_uidlist_sync_ctx, 1); @@ -1430,6 +1550,13 @@ ctx->finished = TRUE; ctx->uidlist->initial_sync = TRUE; + + i_assert(ctx->locked || !ctx->changed); + if ((ctx->changed || ctx->uidlist->recreate) && + !ctx->failed && ctx->locked) T_BEGIN { + if (maildir_uidlist_sync_update(ctx) < 0) + ctx->failed = TRUE; + } T_END; } int maildir_uidlist_sync_deinit(struct maildir_uidlist_sync_ctx **_ctx) @@ -1441,18 +1568,8 @@ if (!ctx->finished) maildir_uidlist_sync_finish(ctx); - if (ctx->partial) maildir_uidlist_mark_all(ctx->uidlist, FALSE); - - i_assert(ctx->locked || !ctx->changed); - if ((ctx->changed || ctx->uidlist->recreate) && - !ctx->failed && ctx->locked) { - T_BEGIN { - ret = maildir_uidlist_sync_update(ctx); - } T_END; - } - if (ctx->locked) maildir_uidlist_unlock(ctx->uidlist);
--- a/src/lib-storage/index/maildir/maildir-uidlist.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/maildir/maildir-uidlist.h Mon Jun 09 05:11:18 2008 +0300 @@ -11,7 +11,8 @@ MAILDIR_UIDLIST_SYNC_PARTIAL = 0x01, MAILDIR_UIDLIST_SYNC_KEEP_STATE = 0x02, MAILDIR_UIDLIST_SYNC_FORCE = 0x04, - MAILDIR_UIDLIST_SYNC_TRYLOCK = 0x08 + MAILDIR_UIDLIST_SYNC_TRYLOCK = 0x08, + MAILDIR_UIDLIST_SYNC_NOREFRESH = 0x10 }; enum maildir_uidlist_rec_flag { @@ -55,6 +56,9 @@ and storage has NFS_FLUSH flag set, the NFS attribute cache is flushed to make sure that we see the latest uidlist file. */ int maildir_uidlist_refresh(struct maildir_uidlist *uidlist); +/* Like maildir_uidlist_refresh(), but if uidlist isn't opened yet, try to + fill in the uidvalidity/nextuid from index file instead. */ +int maildir_uidlist_refresh_fast_init(struct maildir_uidlist *uidlist); /* Returns uidlist record for given filename, or NULL if not found. */ const char * @@ -76,6 +80,7 @@ void maildir_uidlist_set_next_uid(struct maildir_uidlist *uidlist, uint32_t next_uid, bool force); +/* Update extended record. value=NULL removes the key. */ void maildir_uidlist_set_ext(struct maildir_uidlist *uidlist, uint32_t uid, enum maildir_uidlist_rec_ext_key key, const char *value);
--- a/src/lib-storage/index/mbox/istream-raw-mbox.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/mbox/istream-raw-mbox.c Mon Jun 09 05:11:18 2008 +0300 @@ -238,15 +238,19 @@ FIXME: if From-line is longer than input buffer, we break. probably irrelevant.. */ i++; - from_after_pos = i; - from_start_pos = i - 6; - if (from_start_pos > 0 && - buf[from_start_pos-1] == '\r') { - /* CR also belongs to it. */ - crlf_ending = TRUE; - from_start_pos--; - } else { - crlf_ending = FALSE; + if (rstream->hdr_offset + rstream->mail_size == + stream->istream.v_offset + i - 6 || + rstream->mail_size == (uoff_t)-1) { + from_after_pos = i; + from_start_pos = i - 6; + if (from_start_pos > 0 && + buf[from_start_pos-1] == '\r') { + /* CR also belongs to it. */ + crlf_ending = TRUE; + from_start_pos--; + } else { + crlf_ending = FALSE; + } } fromp = mbox_from; } else if (from_start_pos != (size_t)-1) { @@ -290,6 +294,17 @@ new_pos--; } + if (stream->istream.v_offset - + rstream->hdr_offset + new_pos > rstream->mail_size) { + /* istream_raw_mbox_set_next_offset() used invalid + cached next_offset? */ + i_error("Next message unexpectedly lost from %"PRIuUOFF_T, + rstream->hdr_offset + rstream->mail_size); + rstream->eof = TRUE; + rstream->corrupted = TRUE; + return -1; + } + stream->buffer = buf; if (new_pos == stream->pos) { if (stream->istream.eof || ret > 0) @@ -379,8 +394,7 @@ char *sender; /* minimal: "From x Thu Nov 29 22:33:52 2001" = 31 chars */ - if (i_stream_read_data(rstream->istream.parent, &data, &size, 30) == -1) - return -1; + (void)i_stream_read_data(rstream->istream.parent, &data, &size, 30); if ((size == 1 && data[0] == '\n') || (size == 2 && data[0] == '\r' && data[1] == '\n')) { @@ -469,33 +483,41 @@ return rstream->body_offset; } -uoff_t istream_raw_mbox_get_body_size(struct istream *stream, uoff_t body_size) +uoff_t istream_raw_mbox_get_body_size(struct istream *stream, + uoff_t expected_body_size) { struct raw_mbox_istream *rstream = (struct raw_mbox_istream *)stream->real_stream; const unsigned char *data; size_t size; - uoff_t old_offset; + uoff_t old_offset, body_size; i_assert(rstream->hdr_offset != (uoff_t)-1); i_assert(rstream->body_offset != (uoff_t)-1); - if (rstream->mail_size != (uoff_t)-1) { - return rstream->mail_size - - (rstream->body_offset - rstream->hdr_offset); - } - + body_size = rstream->mail_size == (uoff_t)-1 ? (uoff_t)-1 : + rstream->mail_size - (rstream->body_offset - + rstream->hdr_offset); old_offset = stream->v_offset; - if (body_size != (uoff_t)-1) { + if (expected_body_size != (uoff_t)-1) { + /* if we already have the existing body size, use it as long as + it's >= expected body_size. otherwise the previous parsing + may have stopped at a From_-line that belongs to the body. */ + if (body_size != (uoff_t)-1 && body_size >= expected_body_size) + return body_size; + i_stream_seek(rstream->istream.parent, - rstream->body_offset + body_size); + rstream->body_offset + expected_body_size); if (istream_raw_mbox_is_valid_from(rstream) > 0) { - rstream->mail_size = body_size + + rstream->mail_size = expected_body_size + (rstream->body_offset - rstream->hdr_offset); i_stream_seek(stream, old_offset); - return body_size; + return expected_body_size; } + /* invalid expected_body_size */ } + if (body_size != (uoff_t)-1) + return body_size; /* have to read through the message body */ while (i_stream_read_data(stream, &data, &size, 0) > 0) @@ -535,12 +557,13 @@ return rstream->crlf_ending; } -void istream_raw_mbox_next(struct istream *stream, uoff_t body_size) +void istream_raw_mbox_next(struct istream *stream, uoff_t expected_body_size) { struct raw_mbox_istream *rstream = (struct raw_mbox_istream *)stream->real_stream; + uoff_t body_size; - body_size = istream_raw_mbox_get_body_size(stream, body_size); + body_size = istream_raw_mbox_get_body_size(stream, expected_body_size); rstream->mail_size = (uoff_t)-1; rstream->received_time = rstream->next_received_time; @@ -606,6 +629,16 @@ return rstream->corrupted ? -1 : 0; } +void istream_raw_mbox_set_next_offset(struct istream *stream, uoff_t offset) +{ + struct raw_mbox_istream *rstream = + (struct raw_mbox_istream *)stream->real_stream; + + i_assert(rstream->hdr_offset != (uoff_t)-1); + + rstream->mail_size = offset - rstream->hdr_offset; +} + bool istream_raw_mbox_is_eof(struct istream *stream) { struct raw_mbox_istream *rstream =
--- a/src/lib-storage/index/mbox/istream-raw-mbox.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/mbox/istream-raw-mbox.h Mon Jun 09 05:11:18 2008 +0300 @@ -12,10 +12,11 @@ /* Return offset to beginning of the body. */ uoff_t istream_raw_mbox_get_body_offset(struct istream *stream); -/* Return the number of bytes in the body of this message. If body_size isn't - (uoff_t)-1, we'll use it as potentially valid body size to avoid actually - reading through the whole message. */ -uoff_t istream_raw_mbox_get_body_size(struct istream *stream, uoff_t body_size); +/* Return the number of bytes in the body of this message. If + expected_body_size isn't (uoff_t)-1, we'll use it as potentially valid body + size to avoid actually reading through the whole message. */ +uoff_t istream_raw_mbox_get_body_size(struct istream *stream, + uoff_t expected_body_size); /* Return received time of current message, or (time_t)-1 if the timestamp is broken. */ @@ -26,14 +27,18 @@ /* Return TRUE if the empty line between this and the next mail contains CR. */ bool istream_raw_mbox_has_crlf_ending(struct istream *stream); -/* Jump to next message. If body_size isn't (uoff_t)-1, we'll use it as - potentially valid body size. */ -void istream_raw_mbox_next(struct istream *stream, uoff_t body_size); +/* Jump to next message. If expected_body_size isn't (uoff_t)-1, we'll use it + as potentially valid body size. */ +void istream_raw_mbox_next(struct istream *stream, uoff_t expected_body_size); /* Seek to message at given offset. offset must point to beginning of "\nFrom ", or 0 for beginning of file. Returns -1 if it offset doesn't contain a valid From-line. */ int istream_raw_mbox_seek(struct istream *stream, uoff_t offset); +/* Set next message's start offset. If this isn't set, read stops at the next + valid From_-line, even if it belongs to the current message's body + (Content-Length: header can be used to determine that). */ +void istream_raw_mbox_set_next_offset(struct istream *stream, uoff_t offset); /* Returns TRUE if we've read the whole mbox. */ bool istream_raw_mbox_is_eof(struct istream *stream);
--- a/src/lib-storage/index/mbox/mbox-lock.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/mbox/mbox-lock.c Mon Jun 09 05:11:18 2008 +0300 @@ -533,8 +533,15 @@ /* non-blocking lock trying failed */ return 0; } - mbox_set_syscall_error(ctx->mbox, "fcntl()"); alarm(0); + if (errno != EACCES) { + mbox_set_syscall_error(ctx->mbox, "fcntl()"); + return -1; + } + mail_storage_set_critical(&ctx->mbox->storage->storage, + "fcntl() failed with mbox file %s: " + "File is locked by another process (EACCES)", + ctx->mbox->path); return -1; }
--- a/src/lib-storage/index/mbox/mbox-mail.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/mbox/mbox-mail.c Mon Jun 09 05:11:18 2008 +0300 @@ -166,12 +166,52 @@ return index_mail_get_special(_mail, field, value_r); } +static bool +mbox_mail_get_next_offset(struct index_mail *mail, uoff_t *next_offset_r) +{ + struct mbox_mailbox *mbox = (struct mbox_mailbox *)mail->ibox; + struct mail_index_view *view; + const struct mail_index_header *hdr; + uint32_t seq; + int trailer_size; + bool ret; + + hdr = mail_index_get_header(mail->trans->trans_view); + if (mail->mail.mail.seq > hdr->messages_count) { + /* we're appending a new message */ + return FALSE; + } + + /* We can't really trust trans_view. The next message may already be + expugned from it. hdr.sync_size may also be updated, but + hdr.messages_count not. So refresh the index to get the latest + changes and get the next message's offset using a new view. */ + (void)mail_index_refresh(mail->ibox->index); + + view = mail_index_view_open(mail->ibox->index); + hdr = mail_index_get_header(view); + if (!mail_index_lookup_seq(view, mail->mail.mail.uid, &seq)) + i_panic("Message unexpectedly expunged from index"); + + if (seq == hdr->messages_count) { + /* last message, use the synced mbox size */ + trailer_size = (mbox->storage->storage.flags & + MAIL_STORAGE_FLAG_SAVE_CRLF) != 0 ? 2 : 1; + *next_offset_r = hdr->sync_size - trailer_size; + ret = TRUE; + } else { + ret = mbox_file_lookup_offset(mbox, view, seq + 1, + next_offset_r) > 0; + } + mail_index_view_close(&view); + return ret; +} + static int mbox_mail_get_physical_size(struct mail *_mail, uoff_t *size_r) { struct index_mail *mail = (struct index_mail *)_mail; struct index_mail_data *data = &mail->data; struct mbox_mailbox *mbox = (struct mbox_mailbox *)mail->ibox; - const struct mail_index_header *hdr; struct istream *input; struct message_size hdr_size; uoff_t old_offset, body_offset, body_size, next_offset; @@ -194,26 +234,10 @@ /* use the next message's offset to avoid reading through the entire message body to find out its size */ - hdr = mail_index_get_header(mail->trans->trans_view); - if (_mail->seq >= hdr->messages_count) { - if (_mail->seq == hdr->messages_count) { - /* last message, use the synced mbox size */ - int trailer_size; - - trailer_size = (mbox->storage->storage.flags & - MAIL_STORAGE_FLAG_SAVE_CRLF) != 0 ? - 2 : 1; - body_size = hdr->sync_size - body_offset - trailer_size; - } else { - /* we're appending a new message */ - body_size = (uoff_t)-1; - } - } else if (mbox_file_lookup_offset(mbox, mail->trans->trans_view, - _mail->seq + 1, &next_offset) > 0) { + if (mbox_mail_get_next_offset(mail, &next_offset)) body_size = next_offset - body_offset; - } else { + else body_size = (uoff_t)-1; - } /* verify that the calculated body size is correct */ body_size = istream_raw_mbox_get_body_size(mbox->mbox_stream, @@ -226,31 +250,53 @@ return 0; } +static int mbox_mail_init_stream(struct index_mail *mail) +{ + struct mbox_mailbox *mbox = (struct mbox_mailbox *)mail->ibox; + struct istream *raw_stream; + uoff_t hdr_offset, next_offset; + + if (mbox_mail_seek(mail) < 0) + return -1; + + if (!mbox_mail_get_next_offset(mail, &next_offset)) { + if (mbox_mail_seek(mail) < 0) + return -1; + if (!mbox_mail_get_next_offset(mail, &next_offset)) { + i_warning("mbox %s: Can't find next message offset " + "for uid=%u", + mbox->path, mail->mail.mail.uid); + next_offset = (uoff_t)-1; + } + } + + raw_stream = mbox->mbox_stream; + hdr_offset = istream_raw_mbox_get_header_offset(raw_stream); + i_stream_seek(raw_stream, hdr_offset); + + if (next_offset != (uoff_t)-1) + istream_raw_mbox_set_next_offset(raw_stream, next_offset); + + raw_stream = i_stream_create_limit(raw_stream, (uoff_t)-1); + mail->data.stream = + i_stream_create_header_filter(raw_stream, + HEADER_FILTER_EXCLUDE | HEADER_FILTER_NO_CR, + mbox_hide_headers, mbox_hide_headers_count, + null_header_filter_callback, NULL); + i_stream_unref(&raw_stream); + return 0; +} + static int mbox_mail_get_stream(struct mail *_mail, struct message_size *hdr_size, struct message_size *body_size, struct istream **stream_r) { struct index_mail *mail = (struct index_mail *)_mail; - struct index_mail_data *data = &mail->data; - struct mbox_mailbox *mbox = (struct mbox_mailbox *)mail->ibox; - struct istream *raw_stream; - uoff_t offset; - if (data->stream == NULL) { - if (mbox_mail_seek(mail) < 0) + if (mail->data.stream == NULL) { + if (mbox_mail_init_stream(mail) < 0) return -1; - - raw_stream = mbox->mbox_stream; - offset = istream_raw_mbox_get_header_offset(raw_stream); - i_stream_seek(raw_stream, offset); - raw_stream = i_stream_create_limit(raw_stream, (uoff_t)-1); - data->stream = - i_stream_create_header_filter(raw_stream, - HEADER_FILTER_EXCLUDE | HEADER_FILTER_NO_CR, - mbox_hide_headers, mbox_hide_headers_count, - null_header_filter_callback, NULL); - i_stream_unref(&raw_stream); } return index_mail_init_stream(mail, hdr_size, body_size, stream_r);
--- a/src/lib-storage/index/mbox/mbox-save.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/mbox/mbox-save.c Mon Jun 09 05:11:18 2008 +0300 @@ -328,6 +328,7 @@ ctx->output = o_stream_create_fd_file(mbox->mbox_fd, ctx->append_offset, FALSE); + o_stream_cork(ctx->output); } return 0; } @@ -381,7 +382,8 @@ /* filter out unwanted headers and keep track of headers' MD5 sum */ filter = i_stream_create_header_filter(input, HEADER_FILTER_EXCLUDE | - HEADER_FILTER_NO_CR, + HEADER_FILTER_NO_CR | + HEADER_FILTER_ADD_MISSING_EOH, mbox_save_drop_headers, mbox_save_drop_headers_count, save_header_callback, ctx); @@ -652,6 +654,10 @@ { struct mbox_save_context *ctx = (struct mbox_save_context *)_ctx; + /* make sure everything is written */ + if (o_stream_flush(ctx->output) < 0) + return write_error(ctx); + ctx->finished = TRUE; if (!ctx->failed) { T_BEGIN { @@ -670,8 +676,10 @@ if (ctx->failed && ctx->mail_offset != (uoff_t)-1) { /* saving this mail failed - truncate back to beginning of it */ + (void)o_stream_flush(ctx->output); if (ftruncate(ctx->mbox->mbox_fd, (off_t)ctx->mail_offset) < 0) mbox_set_syscall_error(ctx->mbox, "ftruncate()"); + o_stream_seek(ctx->output, ctx->mail_offset); ctx->mail_offset = (uoff_t)-1; } @@ -720,16 +728,20 @@ if (!ctx->mbox->mbox_sync_dirty && ret == 0) { uint32_t sync_stamp = st.st_mtime; - uint64_t sync_size = st.st_size; mail_index_update_header(ctx->trans, offsetof(struct mail_index_header, sync_stamp), &sync_stamp, sizeof(sync_stamp), TRUE); + } + if (ret == 0) { + /* sync_size is used in calculating the last message's + size. it must be up-to-date. */ + uint64_t sync_size = st.st_size; + mail_index_update_header(ctx->trans, offsetof(struct mail_index_header, sync_size), &sync_size, sizeof(sync_size), TRUE); } - *t->ictx.last_saved_uid = ctx->next_uid - 1; }
--- a/src/lib-storage/index/mbox/mbox-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/mbox/mbox-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -4,6 +4,7 @@ #include "ioloop.h" #include "array.h" #include "istream.h" +#include "restrict-access.h" #include "mkdir-parents.h" #include "unlink-directory.h" #include "home-expand.h" @@ -71,6 +72,7 @@ static int mbox_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name, enum mailbox_list_file_type type, enum mailbox_info_flags *flags); static int mbox_list_delete_mailbox(struct mailbox_list *list, @@ -471,6 +473,12 @@ /* make sure inbox file itself exists */ fd = open(inbox_path, O_RDWR | O_CREAT | O_EXCL, 0660); + if (fd == -1 && errno == EACCES) { + /* try again with increased privileges */ + (void)restrict_access_use_priv_gid(); + fd = open(inbox_path, O_RDWR | O_CREAT | O_EXCL, 0660); + restrict_access_drop_priv_gid(); + } if (fd != -1) (void)close(fd); else if (errno == ENOTDIR && @@ -797,6 +805,7 @@ static int mbox_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name ATTR_UNUSED, enum mailbox_list_file_type type, enum mailbox_info_flags *flags_r) {
--- a/src/lib-storage/index/mbox/mbox-sync-private.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/mbox/mbox-sync-private.h Mon Jun 09 05:11:18 2008 +0300 @@ -143,6 +143,7 @@ unsigned int moved_offsets:1; unsigned int ext_modified:1; unsigned int index_reset:1; + unsigned int errors:1; }; int mbox_sync(struct mbox_mailbox *mbox, enum mbox_sync_flags flags);
--- a/src/lib-storage/index/mbox/mbox-sync.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/mbox/mbox-sync.c Mon Jun 09 05:11:18 2008 +0300 @@ -68,6 +68,7 @@ { va_list va; + sync_ctx->errors = TRUE; if (sync_ctx->ext_modified) { mail_storage_set_critical(&sync_ctx->mbox->storage->storage, "mbox file %s was modified while we were syncing, " @@ -1410,8 +1411,8 @@ &sync_size, sizeof(sync_size), TRUE); } - first_recent_uid = !sync_ctx->mbox->ibox.keep_recent ? 0 : - sync_ctx->last_nonrecent_uid + 1; + first_recent_uid = !sync_ctx->mbox->ibox.keep_recent ? + sync_ctx->next_uid : sync_ctx->last_nonrecent_uid + 1; if (sync_ctx->hdr->first_recent_uid < first_recent_uid) { mail_index_update_header(sync_ctx->t, offsetof(struct mail_index_header, first_recent_uid), @@ -1441,6 +1442,7 @@ mail_index_reset(sync_ctx->t); sync_ctx->reset_hdr.next_uid = 1; sync_ctx->hdr = &sync_ctx->reset_hdr; + index_mailbox_reset_uidvalidity(&sync_ctx->mbox->ibox); } sync_ctx->prev_msg_uid = 0; @@ -1454,6 +1456,7 @@ sync_ctx->dest_first_mail = TRUE; sync_ctx->ext_modified = FALSE; + sync_ctx->errors = FALSE; } static int mbox_sync_do(struct mbox_sync_context *sync_ctx, @@ -1500,16 +1503,25 @@ } mbox_sync_restart(sync_ctx); - for (i = 0; i < 3; i++) { + for (i = 0;;) { ret = mbox_sync_loop(sync_ctx, &mail_ctx, partial); - if (ret > 0) + if (ret > 0 && !sync_ctx->errors) break; if (ret < 0) return -1; - /* partial syncing didn't work, do it again. we get here - also if we ran out of UIDs. */ - i_assert(sync_ctx->mbox->mbox_sync_dirty); + /* a) partial sync didn't work + b) we ran out of UIDs + c) syncing had errors */ + if (sync_ctx->delay_writes && !sync_ctx->mbox->mbox_readonly && + (sync_ctx->errors || sync_ctx->renumber_uids)) { + /* fixing a broken mbox state, be sure to write + the changes. */ + sync_ctx->delay_writes = FALSE; + } + if (++i == 3) + break; + mbox_sync_restart(sync_ctx); partial = FALSE; } @@ -1522,6 +1534,14 @@ ignore them, as we've overwritten them above. */ index_sync_changes_reset(sync_ctx->sync_changes); + if (sync_ctx->base_uid_last != sync_ctx->next_uid-1 && + ret == 0 && !sync_ctx->delay_writes && + sync_ctx->base_uid_last_offset != 0) { + /* Rewrite uid_last in X-IMAPbase header if we've seen it + (ie. the file isn't empty) */ + ret = mbox_rewrite_base_uid_last(sync_ctx); + } + if (mbox_sync_update_index_header(sync_ctx) < 0) return -1; @@ -1644,7 +1664,6 @@ before index syncing is started to avoid deadlocks, so we don't have much choice either (well, easy ones anyway). */ int lock_type = mbox->mbox_readonly ? F_RDLCK : F_WRLCK; - int ret; if ((ret = mbox_lock(mbox, lock_type, &lock_id)) <= 0) { if (ret == 0 || lock_type == F_RDLCK) @@ -1714,7 +1733,7 @@ sync_ctx.sync_view = sync_view; sync_ctx.t = trans; sync_ctx.mail_keyword_pool = - pool_alloconly_create("mbox keywords", 256); + pool_alloconly_create("mbox keywords", 512); sync_ctx.saved_keywords_pool = pool_alloconly_create("mbox saved keywords", 4096); @@ -1780,15 +1799,8 @@ sync_ctx.t = NULL; sync_ctx.index_sync_ctx = NULL; - if (sync_ctx.base_uid_last != sync_ctx.next_uid-1 && - ret == 0 && !sync_ctx.delay_writes && - sync_ctx.base_uid_last_offset != 0) { - /* Rewrite uid_last in X-IMAPbase header if we've seen it - (ie. the file isn't empty) */ - ret = mbox_rewrite_base_uid_last(&sync_ctx); - } - - if (ret == 0 && mbox->mbox_fd != -1 && mbox->ibox.keep_recent) { + if (ret == 0 && mbox->mbox_fd != -1 && mbox->ibox.keep_recent && + !sync_ctx.mbox->mbox_readonly) { /* try to set atime back to its original value */ struct utimbuf buf; struct stat st;
--- a/src/lib-storage/index/raw/raw-mail.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/raw/raw-mail.c Mon Jun 09 05:11:18 2008 +0300 @@ -98,6 +98,9 @@ case MAIL_FETCH_FROM_ENVELOPE: *value_r = mbox->envelope_sender; return 0; + case MAIL_FETCH_UIDL_FILE_NAME: + *value_r = mbox->have_filename ? mbox->path : ""; + return 0; default: return index_mail_get_special(_mail, field, value_r); }
--- a/src/lib-storage/index/raw/raw-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/raw/raw-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -21,6 +21,7 @@ static int raw_list_delete_mailbox(struct mailbox_list *list, const char *name); static int raw_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name, enum mailbox_list_file_type type, enum mailbox_info_flags *flags); @@ -161,8 +162,10 @@ if (stream) mbox->mtime = mbox->ctime = ioloop_time; - else + else { mbox->mtime = mbox->ctime = (time_t)-1; + mbox->have_filename = TRUE; + } mbox->size = (uoff_t)-1; index_storage_mailbox_init(&mbox->ibox, name, flags, FALSE); @@ -200,6 +203,7 @@ static int raw_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name ATTR_UNUSED, enum mailbox_list_file_type type, enum mailbox_info_flags *flags_r) {
--- a/src/lib-storage/index/raw/raw-storage.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/index/raw/raw-storage.h Mon Jun 09 05:11:18 2008 +0300 @@ -24,6 +24,7 @@ const char *envelope_sender; unsigned int synced:1; + unsigned int have_filename:1; }; extern struct mail_vfuncs raw_mail_vfuncs;
--- a/src/lib-storage/list/mailbox-list-fs-iter.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/list/mailbox-list-fs-iter.c Mon Jun 09 05:11:18 2008 +0300 @@ -349,7 +349,7 @@ inbox_path = mailbox_list_get_path(ctx->ctx.list, "INBOX", MAILBOX_LIST_PATH_TYPE_DIR); path_split(inbox_path, &dir, &fname); - if (ctx->ctx.list->v.iter_is_mailbox(&ctx->ctx, dir, fname, + if (ctx->ctx.list->v.iter_is_mailbox(&ctx->ctx, dir, fname, "INBOX", MAILBOX_LIST_FILE_TYPE_UNKNOWN, &ctx->info.flags) < 0) ctx->ctx.failed = TRUE; @@ -492,7 +492,7 @@ ctx->info.flags = 0; ret = ctx->ctx.list->v. iter_is_mailbox(&ctx->ctx, ctx->dir->real_path, fname, - entry->type, &ctx->info.flags); + list_path, entry->type, &ctx->info.flags); if (ret <= 0) return ret; @@ -550,7 +550,9 @@ path = mailbox_list_get_path(ctx->ctx.list, ctx->info.name, MAILBOX_LIST_PATH_TYPE_DIR); path_split(path, &dir, &fname); + ctx->info.flags = 0; if (ctx->ctx.list->v.iter_is_mailbox(&ctx->ctx, dir, fname, + ctx->info.name, MAILBOX_LIST_FILE_TYPE_UNKNOWN, &ctx->info.flags) < 0) ctx->ctx.failed = TRUE; @@ -572,9 +574,9 @@ if (dir->dirp != NULL) { if (dir->next_entry != NULL) { - const struct list_dir_entry *ret = dir->next_entry; + const struct list_dir_entry *entry = dir->next_entry; dir->next_entry = NULL; - return ret; + return entry; } d = readdir(dir->dirp); if (d == NULL)
--- a/src/lib-storage/list/mailbox-list-maildir-iter.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/list/mailbox-list-maildir-iter.c Mon Jun 09 05:11:18 2008 +0300 @@ -89,6 +89,33 @@ } } +static void maildir_set_children(struct maildir_list_iterate_context *ctx, + string_t *mailbox) +{ + struct mailbox_node *node; + const char *p, *mailbox_c; + char hierarchy_sep; + + if ((ctx->ctx.flags & MAILBOX_LIST_ITER_VIRTUAL_NAMES) != 0) + hierarchy_sep = ctx->ctx.list->ns->sep; + else + hierarchy_sep = ctx->ctx.list->ns->real_sep; + + /* mark the first existing parent as containing children */ + mailbox_c = str_c(mailbox); + while ((p = strrchr(mailbox_c, hierarchy_sep)) != NULL) { + str_truncate(mailbox, (size_t) (p-mailbox_c)); + mailbox_c = str_c(mailbox); + + node = mailbox_tree_lookup(ctx->tree_ctx, mailbox_c); + if (node != NULL) { + node->flags &= ~MAILBOX_NOCHILDREN; + node->flags |= MAILBOX_CHILDREN; + break; + } + } +} + static int maildir_fill_readdir(struct maildir_list_iterate_context *ctx, struct imap_match_glob *glob, bool update_only) @@ -156,6 +183,7 @@ T_BEGIN { ret = ctx->ctx.list->v. iter_is_mailbox(&ctx->ctx, ctx->dir, fname, + mailbox_name, mailbox_list_get_file_type(d), &flags); } T_END; if (ret <= 0) { @@ -191,6 +219,9 @@ node->flags |= MAILBOX_MATCHED; node->flags |= flags; node_fix_parents(node); + } else { + i_assert(update_only); + maildir_set_children(ctx, mailbox); } } } @@ -221,7 +252,7 @@ imap_match(glob, "INBOX") == IMAP_MATCH_YES) { /* see if INBOX exists. */ ret = ctx->ctx.list->v. - iter_is_mailbox(&ctx->ctx, ctx->dir, "", + iter_is_mailbox(&ctx->ctx, ctx->dir, "", "INBOX", MAILBOX_LIST_FILE_TYPE_UNKNOWN, &flags); if (ret > 0) { node = mailbox_tree_get(ctx->tree_ctx,
--- a/src/lib-storage/list/mailbox-list-subscriptions.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/list/mailbox-list-subscriptions.c Mon Jun 09 05:11:18 2008 +0300 @@ -13,10 +13,10 @@ bool update_only) { struct mail_namespace *ns = ctx->list->ns; + struct mailbox_list_iter_update_context update_ctx; struct subsfile_list_context *subsfile_ctx; const char *path, *name; string_t *vname; - bool match_parents; vname = t_str_new(256); path = t_strconcat(ctx->list->set.control_dir != NULL ? @@ -25,13 +25,19 @@ "/", ctx->list->set.subscription_fname, NULL); subsfile_ctx = subsfile_list_init(ctx->list, path); - match_parents = + memset(&update_ctx, 0, sizeof(update_ctx)); + update_ctx.iter_ctx = ctx; + update_ctx.tree_ctx = tree_ctx; + update_ctx.glob = glob; + update_ctx.leaf_flags = MAILBOX_SUBSCRIBED; + update_ctx.parent_flags = MAILBOX_CHILD_SUBSCRIBED; + update_ctx.update_only = update_only; + update_ctx.match_parents = (ctx->flags & MAILBOX_LIST_ITER_SELECT_RECURSIVEMATCH) != 0; while ((name = subsfile_list_next(subsfile_ctx)) != NULL) { name = mail_namespace_get_vname(ns, vname, name); - mailbox_list_iter_update(ctx, tree_ctx, glob, update_only, - match_parents, name); + mailbox_list_iter_update(&update_ctx, name); } return subsfile_list_deinit(subsfile_ctx); }
--- a/src/lib-storage/mail-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/mail-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -542,7 +542,6 @@ struct mailbox_status *status_r) { struct mailbox_sync_context *ctx; - struct mailbox_sync_rec sync_rec; if (array_count(&box->search_results) == 0) { /* we don't care about mailbox's current state, so we might @@ -551,8 +550,6 @@ } ctx = mailbox_sync_init(box, flags); - while (mailbox_sync_next(ctx, &sync_rec)) - ; return mailbox_sync_deinit(&ctx, status_items, status_r); } @@ -703,9 +700,12 @@ int mailbox_transaction_commit(struct mailbox_transaction_context **t) { - uint32_t tmp; + uint32_t uidvalidity, uid1, uid2; - return mailbox_transaction_commit_get_uids(t, &tmp, &tmp, &tmp); + /* Store the return values to separate temporary variables so that + plugins overriding transaction_commit() can look at them. */ + return mailbox_transaction_commit_get_uids(t, &uidvalidity, + &uid1, &uid2); } int mailbox_transaction_commit_get_uids(struct mailbox_transaction_context **_t,
--- a/src/lib-storage/mail-storage.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/mail-storage.h Mon Jun 09 05:11:18 2008 +0300 @@ -124,7 +124,8 @@ MAIL_FETCH_IMAP_ENVELOPE = 0x00004000, MAIL_FETCH_FROM_ENVELOPE = 0x00008000, MAIL_FETCH_HEADER_MD5 = 0x00010000, - MAIL_FETCH_UIDL_FILE_NAME = 0x00020000 + MAIL_FETCH_UIDL_FILE_NAME = 0x00020000, + MAIL_FETCH_UIDL_BACKEND = 0x00040000 }; enum mailbox_transaction_flags {
--- a/src/lib-storage/mailbox-list-private.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/mailbox-list-private.h Mon Jun 09 05:11:18 2008 +0300 @@ -41,6 +41,7 @@ flags may be updated (especially the children flags). */ int (*iter_is_mailbox)(struct mailbox_list_iterate_context *ctx, const char *dir, const char *fname, + const char *mailbox_name, enum mailbox_list_file_type type, enum mailbox_info_flags *flags_r); @@ -93,6 +94,17 @@ bool failed; }; +struct mailbox_list_iter_update_context { + struct mailbox_list_iterate_context *iter_ctx; + struct mailbox_tree_context *tree_ctx; + + struct imap_match_glob *glob; + enum mailbox_info_flags leaf_flags, parent_flags; + + unsigned int update_only:1; + unsigned int match_parents:1; +}; + /* Modules should use do "my_id = mailbox_list_module_id++" and use objects' module_contexts[id] for their own purposes. */ extern struct mailbox_list_module_register mailbox_list_module_register; @@ -110,10 +122,8 @@ int mailbox_list_delete_index_control(struct mailbox_list *list, const char *name); -void mailbox_list_iter_update(struct mailbox_list_iterate_context *ctx, - struct mailbox_tree_context *tree_ctx, - struct imap_match_glob *glob, bool update_only, - bool match_parents, const char *name); +void mailbox_list_iter_update(struct mailbox_list_iter_update_context *ctx, + const char *name); bool mailbox_list_name_is_too_large(const char *name, char sep); enum mailbox_list_file_type mailbox_list_get_file_type(const struct dirent *d);
--- a/src/lib-storage/mailbox-list.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/mailbox-list.c Mon Jun 09 05:11:18 2008 +0300 @@ -474,38 +474,36 @@ } static void -mailbox_list_iter_update_real(struct mailbox_list_iterate_context *ctx, - struct mailbox_tree_context *tree_ctx, - struct imap_match_glob *glob, bool update_only, - bool match_parents, const char *name) +mailbox_list_iter_update_real(struct mailbox_list_iter_update_context *ctx, + const char *name) { - struct mail_namespace *ns = ctx->list->ns; + struct mail_namespace *ns = ctx->iter_ctx->list->ns; struct mailbox_node *node; - enum mailbox_info_flags create_flags, always_flags; + enum mailbox_info_flags create_flags = 0, always_flags; enum imap_match_result match; const char *p; bool created, add_matched; - create_flags = (update_only || - (ctx->flags & MAILBOX_LIST_ITER_RETURN_NO_FLAGS) == 0) ? - (MAILBOX_NONEXISTENT | MAILBOX_NOCHILDREN) : 0; - always_flags = MAILBOX_SUBSCRIBED; + if (ctx->update_only || + (ctx->iter_ctx->flags & MAILBOX_LIST_ITER_RETURN_NO_FLAGS) == 0) + create_flags = MAILBOX_NONEXISTENT | MAILBOX_NOCHILDREN; + always_flags = ctx->leaf_flags; add_matched = TRUE; for (;;) { created = FALSE; - match = imap_match(glob, name); + match = imap_match(ctx->glob, name); if (match == IMAP_MATCH_YES) { - node = update_only ? - mailbox_tree_lookup(tree_ctx, name) : - mailbox_tree_get(tree_ctx, name, &created); + node = ctx->update_only ? + mailbox_tree_lookup(ctx->tree_ctx, name) : + mailbox_tree_get(ctx->tree_ctx, name, &created); if (created) { node->flags = create_flags; if (create_flags != 0) node_fix_parents(node); } if (node != NULL) { - if (!update_only && add_matched) + if (!ctx->update_only && add_matched) node->flags |= MAILBOX_MATCHED; node->flags |= always_flags; } @@ -521,7 +519,7 @@ return the parent mailbox. */ } - if (!match_parents) + if (!ctx->match_parents) break; /* see if parent matches */ @@ -531,18 +529,15 @@ name = t_strdup_until(name, p); create_flags &= ~MAILBOX_NOCHILDREN; - always_flags = MAILBOX_CHILDREN | MAILBOX_CHILD_SUBSCRIBED; + always_flags = MAILBOX_CHILDREN | ctx->parent_flags; } } -void mailbox_list_iter_update(struct mailbox_list_iterate_context *ctx, - struct mailbox_tree_context *tree_ctx, - struct imap_match_glob *glob, bool update_only, - bool match_parents, const char *name) +void mailbox_list_iter_update(struct mailbox_list_iter_update_context *ctx, + const char *name) { T_BEGIN { - mailbox_list_iter_update_real(ctx, tree_ctx, glob, update_only, - match_parents, name); + mailbox_list_iter_update_real(ctx, name); } T_END; }
--- a/src/lib-storage/mailbox-tree.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib-storage/mailbox-tree.c Mon Jun 09 05:11:18 2008 +0300 @@ -200,7 +200,7 @@ str_truncate(ctx->path_str, ctx->parent_pos); if (ctx->first_child) { ctx->first_child = FALSE; - if (ctx->parent_pos != 0) { + if (node->parent != NULL) { str_append_c(ctx->path_str, ctx->separator); ctx->parent_pos++; }
--- a/src/lib/compat.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/compat.h Mon Jun 09 05:11:18 2008 +0300 @@ -63,17 +63,37 @@ # error I do not know how to compare dev_t #endif -#ifdef HAVE_STAT_TV_NSEC -# define CMP_ST_MTIME(st1, st2) \ +#ifdef HAVE_STAT_XTIM +# define HAVE_ST_NSECS +# define ST_ATIME_NSEC(st) ((unsigned long)(st).st_atim.tv_nsec) +# define ST_MTIME_NSEC(st) ((unsigned long)(st).st_mtim.tv_nsec) +# define ST_CTIME_NSEC(st) ((unsigned long)(st).st_ctim.tv_nsec) +#elif defined (HAVE_STAT_XTIMESPEC) +# define HAVE_ST_NSECS +# define ST_ATIME_NSEC(st) ((unsigned long)(st).st_atimespec.tv_nsec) +# define ST_MTIME_NSEC(st) ((unsigned long)(st).st_mtimespec.tv_nsec) +# define ST_CTIME_NSEC(st) ((unsigned long)(st).st_ctimespec.tv_nsec) +#else +# define ST_ATIME_NSEC(st) 0UL +# define ST_MTIME_NSEC(st) 0UL +# define ST_CTIME_NSEC(st) 0UL +#endif + +#ifdef HAVE_ST_NSECS +/* TRUE if a nanosecond timestamp from struct stat matches another nanosecond. + If nanoseconds aren't supported in struct stat, returns always TRUE (useful + with NFS if some hosts support nanoseconds and others don't). */ +# define ST_NTIMES_EQUAL(ns1, ns2) ((ns1) == (ns2)) +#else +# define ST_NTIMES_EQUAL(ns1, ns2) TRUE +#endif + +#define CMP_ST_MTIME(st1, st2) \ ((st1)->st_mtime == (st2)->st_mtime && \ - (st1)->st_mtim.tv_nsec == (st2)->st_mtim.tv_nsec) -# define CMP_ST_CTIME(st1, st2) \ + ST_NTIMES_EQUAL(ST_MTIME_NSEC(*(st1)), ST_MTIME_NSEC(*(st2)))) +#define CMP_ST_CTIME(st1, st2) \ ((st1)->st_ctime == (st2)->st_ctime && \ - (st1)->st_ctim.tv_nsec == (st2)->st_ctim.tv_nsec) -#else -# define CMP_ST_MTIME(st1, st2) ((st1)->st_mtime == (st2)->st_mtime) -# define CMP_ST_CTIME(st1, st2) ((st1)->st_ctime == (st2)->st_ctime) -#endif + ST_NTIMES_EQUAL(ST_CTIME_NSEC(*(st1)), ST_CTIME_NSEC(*(st2)))) /* strcasecmp(), strncasecmp() */ #ifndef HAVE_STRCASECMP
--- a/src/lib/data-stack.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/data-stack.c Mon Jun 09 05:11:18 2008 +0300 @@ -67,11 +67,38 @@ static bool clean_after_pop = FALSE; static bool outofmem = FALSE; -union { +static union { struct stack_block block; unsigned char data[128]; } outofmem_area; +static void data_stack_last_buffer_reset(void) +{ + if (last_buffer_block != NULL) { +#ifdef DEBUG + const unsigned char *p; + unsigned int i; + + p = STACK_BLOCK_DATA(current_block) + + (current_block->size - current_block->left) + + MEM_ALIGN(sizeof(size_t)) + MEM_ALIGN(last_buffer_size); +#endif + /* reset t_buffer_get() mark - not really needed but makes it + easier to notice if t_malloc()/t_push()/t_pop() is called + between t_buffer_get() and t_buffer_alloc(). + do this before we get to i_panic() to avoid recursive + panics. */ + last_buffer_block = NULL; + +#ifdef DEBUG + for (i = 0; i < SENTRY_COUNT; i++) { + if (p[i] != CLEAR_CHR) + i_panic("t_buffer_get(): buffer overflow"); + } +#endif + } +} + unsigned int t_push(void) { struct stack_frame_block *frame_block; @@ -107,6 +134,7 @@ frame_block->prev = current_frame_block; current_frame_block = frame_block; } + data_stack_last_buffer_reset(); /* mark our current position */ current_frame_block->block[frame_pos] = current_block; @@ -186,7 +214,6 @@ unsigned int t_pop(void) { struct stack_frame_block *frame_block; - int popped_frame_pos; if (unlikely(frame_pos < 0)) i_panic("t_pop() called with empty stack"); @@ -194,6 +221,7 @@ #ifdef DEBUG t_pop_verify(); #endif + data_stack_last_buffer_reset(); /* update the current block */ current_block = current_frame_block->block[frame_pos]; @@ -215,7 +243,6 @@ current_block->next = NULL; } - popped_frame_pos = frame_pos; if (frame_pos > 0) frame_pos--; else { @@ -290,10 +317,7 @@ data_stack_init(); } - /* reset t_buffer_get() mark - not really needed but makes it easier - to notice if t_malloc() is called between t_buffer_get() and - t_buffer_alloc() */ - last_buffer_block = NULL; + data_stack_last_buffer_reset(); /* allocate only aligned amount of memory so alignment comes always properly */
--- a/src/lib/env-util.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/env-util.c Mon Jun 09 05:11:18 2008 +0300 @@ -9,9 +9,10 @@ void env_put(const char *env) { - if (pool == NULL) - pool = pool_alloconly_create("Environment", 2048); - + if (pool == NULL) { + pool = pool_alloconly_create(MEMPOOL_GROWING"Environment", + 2048); + } if (putenv(p_strdup(pool, env)) != 0) i_fatal("putenv(%s) failed: %m", env); }
--- a/src/lib/failures.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/failures.c Mon Jun 09 05:11:18 2008 +0300 @@ -98,7 +98,7 @@ /* wait until we can write more. this can happen at least when writing to terminal, even if fd is blocking. */ ioloop = io_loop_create(); - io = io_add(IO_WRITE, fd, log_fd_flush_stop, ioloop); + io = io_add(fd, IO_WRITE, log_fd_flush_stop, ioloop); io_loop_run(ioloop); io_remove(&io); io_loop_destroy(&ioloop); @@ -138,15 +138,11 @@ return ret; } -void default_fatal_handler(enum log_type type, int status, - const char *format, va_list args) +static void ATTR_NORETURN +default_fatal_finish(enum log_type type, int status) { const char *backtrace; - if (default_handler(failure_log_type_prefixes[type], log_fd, format, - args) < 0 && status == FATAL_DEFAULT) - status = FATAL_LOGWRITE; - if (type == LOG_TYPE_PANIC || status == FATAL_OUTOFMEM) { if (backtrace_get(&backtrace) == 0) i_error("Raw backtrace: %s", backtrace); @@ -158,6 +154,16 @@ failure_exit(status); } +void default_fatal_handler(enum log_type type, int status, + const char *format, va_list args) +{ + if (default_handler(failure_log_type_prefixes[type], log_fd, format, + args) < 0 && status == FATAL_DEFAULT) + status = FATAL_LOGWRITE; + + default_fatal_finish(type, status); +} + void default_error_handler(enum log_type type, const char *format, va_list args) { int fd = type == LOG_TYPE_INFO ? log_info_fd : log_fd; @@ -292,18 +298,11 @@ void i_syslog_fatal_handler(enum log_type type, int status, const char *fmt, va_list args) { - const char *backtrace; if (syslog_handler(LOG_CRIT, type, fmt, args) < 0 && status == FATAL_DEFAULT) status = FATAL_LOGERROR; - if (type == LOG_TYPE_PANIC) { - if (backtrace_get(&backtrace) == 0) - i_error("Raw backtrace: %s", backtrace); - abort(); - } else { - failure_exit(status); - } + default_fatal_finish(type, status); } void i_syslog_error_handler(enum log_type type, const char *fmt, va_list args) @@ -358,7 +357,7 @@ if (*fd == -1) { *fd = STDERR_FILENO; i_snprintf(buf, sizeof(buf), - "Can't open log file %s: %m", path); + "Can't open log file %s: %m\n", path); (void)write_full(STDERR_FILENO, buf, strlen(buf)); failure_exit(FATAL_LOGOPEN); } @@ -412,19 +411,11 @@ i_internal_fatal_handler(enum log_type type, int status, const char *fmt, va_list args) { - const char *backtrace; - if (internal_handler(log_type_internal_chars[type], fmt, args) < 0 && status == FATAL_DEFAULT) status = FATAL_LOGERROR; - if (type == LOG_TYPE_PANIC) { - if (backtrace_get(&backtrace) == 0) - i_error("Raw backtrace: %s", backtrace); - abort(); - } else { - failure_exit(status); - } + default_fatal_finish(type, status); } static void ATTR_FORMAT(2, 0)
--- a/src/lib/file-dotlock.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/file-dotlock.c Mon Jun 09 05:11:18 2008 +0300 @@ -236,8 +236,10 @@ if (lock_info->have_pid) { /* we've local PID. Check if it exists. */ if (kill(pid, 0) == 0 || errno != ESRCH) { - if (pid != getpid()) + if (pid != getpid()) { + /* process exists, don't override */ return 0; + } /* it's us. either we're locking it again, or it's a stale lock file with same pid than us. either way, recreate it.. */ @@ -427,13 +429,15 @@ /* the lock file doesn't exist anymore, don't sleep */ io_loop_destroy(&ioloop); return; - case IO_NOTIFY_DISABLED: + case IO_NOTIFY_NOSUPPORT: /* listening for files not supported */ io_loop_destroy(&ioloop); lock_info->use_io_notify = FALSE; usleep(LOCK_RANDOM_USLEEP_TIME); return; } + /* timeout after a random time even when using notify, since it + doesn't work reliably with e.g. NFS. */ to = timeout_add(LOCK_RANDOM_USLEEP_TIME/1000, dotlock_wait_end, ioloop); io_loop_run(ioloop);
--- a/src/lib/file-lock.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/file-lock.c Mon Jun 09 05:11:18 2008 +0300 @@ -39,6 +39,7 @@ i_fatal("fcntl() locks not supported"); #else struct flock fl; + const char *errstr; fl.l_type = lock_type; fl.l_whence = SEEK_SET; @@ -64,10 +65,12 @@ errno = EAGAIN; return 0; } - i_error("fcntl(%s) locking failed for file %s: %m", + errstr = errno != EACCES ? strerror(errno) : + "File is locked by another process (EACCES)"; + i_error("fcntl(%s) locking failed for file %s: %s", lock_type == F_UNLCK ? "unlock" : lock_type == F_RDLCK ? "read-lock" : "write-lock", - path); + path, errstr); return -1; #endif }
--- a/src/lib/ioloop-notify-dn.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/ioloop-notify-dn.c Mon Jun 09 05:11:18 2008 +0300 @@ -29,6 +29,21 @@ static struct ioloop_notify_handler_context *io_loop_notify_handler_init(void); +static void ioloop_dnotify_disable(struct ioloop_notify_handler_context *ctx) +{ + if (ctx->disabled) + return; + + if (--sigrt_refcount == 0) + signal(SIGRTMIN, SIG_IGN); + + if (close(ctx->event_pipe[0]) < 0) + i_error("close(dnotify pipe[0]) failed: %m"); + if (close(ctx->event_pipe[1]) < 0) + i_error("close(dnotify pipe[1]) failed: %m"); + ctx->disabled = TRUE; +} + static void sigrt_handler(int signo ATTR_UNUSED, siginfo_t *si, void *data ATTR_UNUSED) { @@ -37,9 +52,14 @@ int saved_errno = errno; int ret; + if (ctx->disabled) + return; + ret = write(ctx->event_pipe[1], &si->si_fd, sizeof(int)); - if (ret < 0 && errno != EINTR && errno != EAGAIN) - i_fatal("write(event_pipe) failed: %m"); + if (ret < 0 && errno != EINTR && errno != EAGAIN) { + i_error("write(dnotify pipe) failed: %m"); + ioloop_dnotify_disable(ctx); + } i_assert(ret <= 0 || ret == sizeof(int)); @@ -55,9 +75,9 @@ ret = read(ctx->event_pipe[0], fd_buf, sizeof(fd_buf)); if (ret < 0) - i_fatal("read(event_pipe) failed: %m"); + i_fatal("read(dnotify pipe) failed: %m"); if ((ret % sizeof(fd_buf[0])) != 0) - i_fatal("read(event_pipe) returned %d", ret); + i_fatal("read(dnotify pipe) returned %d", ret); ret /= sizeof(fd_buf[0]); if (gettimeofday(&ioloop_timeval, &ioloop_timezone) < 0) @@ -77,14 +97,14 @@ { struct ioloop_notify_handler_context *ctx = current_ioloop->notify_handler_context; - int fd, ret; + int fd; *io_r = NULL; if (ctx == NULL) ctx = io_loop_notify_handler_init(); if (ctx->disabled) - return IO_NOTIFY_DISABLED; + return IO_NOTIFY_NOSUPPORT; fd = open(path, O_RDONLY); if (fd == -1) { @@ -99,26 +119,24 @@ /* EINVAL means there's no realtime signals and no dnotify */ if (errno != EINVAL) i_error("fcntl(F_SETSIG) failed: %m"); - ctx->disabled = TRUE; + ioloop_dnotify_disable(ctx); (void)close(fd); - return IO_NOTIFY_DISABLED; + return IO_NOTIFY_NOSUPPORT; } if (fcntl(fd, F_NOTIFY, DN_CREATE | DN_DELETE | DN_RENAME | DN_MULTISHOT) < 0) { if (errno == ENOTDIR) { /* we're trying to add dnotify to a non-directory fd. fail silently. */ - ret = IO_NOTIFY_NOTFOUND; } else { /* dnotify not in kernel. disable it. */ if (errno != EINVAL) i_error("fcntl(F_NOTIFY) failed: %m"); - ctx->disabled = TRUE; - ret = IO_NOTIFY_DISABLED; + ioloop_dnotify_disable(ctx); } (void)fcntl(fd, F_SETSIG, 0); (void)close(fd); - return ret; + return IO_NOTIFY_NOSUPPORT; } if (ctx->event_io == NULL) { @@ -181,7 +199,7 @@ if (errno == EINVAL) { /* kernel is too old to understand even RT signals, so there's no way dnotify works */ - ctx->disabled = TRUE; + ioloop_dnotify_disable(ctx); } else { i_fatal("sigaction(SIGRTMIN) failed: %m"); } @@ -195,14 +213,7 @@ struct ioloop_notify_handler_context *ctx = ioloop->notify_handler_context; - if (--sigrt_refcount == 0) - signal(SIGRTMIN, SIG_IGN); - - if (close(ctx->event_pipe[0]) < 0) - i_error("close(event_pipe[0]) failed: %m"); - if (close(ctx->event_pipe[1]) < 0) - i_error("close(event_pipe[1]) failed: %m"); - + ioloop_dnotify_disable(ctx); i_free(ctx); }
--- a/src/lib/ioloop-notify-inotify.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/ioloop-notify-inotify.c Mon Jun 09 05:11:18 2008 +0300 @@ -95,7 +95,7 @@ if (ctx == NULL) ctx = io_loop_notify_handler_init(); if (ctx->disabled) - return IO_NOTIFY_DISABLED; + return IO_NOTIFY_NOSUPPORT; wd = inotify_add_watch(ctx->inotify_fd, path, IN_CREATE | IN_DELETE | IN_DELETE_SELF | @@ -106,8 +106,9 @@ if (errno == ENOENT || errno == ESTALE) return IO_NOTIFY_NOTFOUND; + i_error("inotify_add_watch(%s) failed: %m", path); ctx->disabled = TRUE; - return IO_NOTIFY_DISABLED; + return IO_NOTIFY_NOSUPPORT; } if (ctx->event_io == NULL) { @@ -153,7 +154,8 @@ i_error("inotify_init() failed: %m"); else { i_warning("Inotify instance limit for user exceeded, " - "disabling."); + "disabling. Increase " + "/proc/sys/fs/inotify/max_user_instances"); } ctx->disabled = TRUE; } else {
--- a/src/lib/ioloop-notify-kqueue.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/ioloop-notify-kqueue.c Mon Jun 09 05:11:18 2008 +0300 @@ -118,7 +118,9 @@ fd = open(path, O_RDONLY); if (fd == -1) { - if (errno != ENOENT) + /* ESTALE could happen with NFS. Don't bother giving an error + message then. */ + if (errno != ENOENT && errno != ESTALE) i_error("open(%s) for kq notify failed: %m", path); return IO_NOTIFY_NOTFOUND; } @@ -140,7 +142,7 @@ i_error("kevent(%d, %s) for notify failed: %m", fd, path); (void)close(fd); i_free(io); - return IO_NOTIFY_DISABLED; + return IO_NOTIFY_NOSUPPORT; } if (ctx->event_io == NULL) {
--- a/src/lib/ioloop-notify-none.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/ioloop-notify-none.c Mon Jun 09 05:11:18 2008 +0300 @@ -12,7 +12,7 @@ void *context ATTR_UNUSED, struct io **io_r) { *io_r = NULL; - return IO_NOTIFY_DISABLED; + return IO_NOTIFY_NOSUPPORT; } void io_loop_notify_remove(struct ioloop *ioloop ATTR_UNUSED,
--- a/src/lib/ioloop.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/ioloop.c Mon Jun 09 05:11:18 2008 +0300 @@ -144,9 +144,16 @@ /* if we came here from io_loop_handle_timeouts(), next_run must be larger than tv_now or we could go to infinite loop */ - if (++timeout->next_run.tv_usec == 0) + timeout->next_run.tv_usec += 1000; + if (timeout->next_run.tv_usec >= 1000000) { timeout->next_run.tv_sec++; + timeout->next_run.tv_usec -= 1000000; + } } + i_assert(tv_now == NULL || + timeout->next_run.tv_sec > tv_now->tv_sec || + (timeout->next_run.tv_sec == tv_now->tv_sec && + timeout->next_run.tv_usec > tv_now->tv_usec)); priorityq_remove(current_ioloop->timeouts, &timeout->item); priorityq_add(current_ioloop->timeouts, &timeout->item); }
--- a/src/lib/ioloop.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/ioloop.h Mon Jun 09 05:11:18 2008 +0300 @@ -20,9 +20,13 @@ }; enum io_notify_result { + /* Notify added successfully */ IO_NOTIFY_ADDED, + /* Specified file doesn't exist, can't wait on it */ IO_NOTIFY_NOTFOUND, - IO_NOTIFY_DISABLED + /* Can't add notify for specified file. Main reasons for this: + a) No notify support at all, b) Only directory notifies supported */ + IO_NOTIFY_NOSUPPORT }; typedef void io_callback_t(void *context);
--- a/src/lib/mempool-alloconly.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/mempool-alloconly.c Mon Jun 09 05:11:18 2008 +0300 @@ -40,10 +40,18 @@ #define SIZEOF_POOLBLOCK (MEM_ALIGN(sizeof(struct pool_block))) #define POOL_BLOCK_DATA(block) \ - ((char *) (block) + SIZEOF_POOLBLOCK) + ((unsigned char *) (block) + SIZEOF_POOLBLOCK) #define DEFAULT_BASE_SIZE MEM_ALIGN(sizeof(struct alloconly_pool)) +#ifdef DEBUG +# define CLEAR_CHR 0xde +# define SENTRY_COUNT 8 +#else +# define SENTRY_COUNT 0 +# define CLEAR_CHR 0 +#endif + static const char *pool_alloconly_get_name(pool_t pool); static void pool_alloconly_ref(pool_t pool); static void pool_alloconly_unref(pool_t *pool); @@ -79,28 +87,48 @@ }; #ifdef DEBUG -static void check_nuls(struct pool_block *block) +static void check_sentries(struct pool_block *block) { - const char *data = POOL_BLOCK_DATA(block); - size_t i; + const unsigned char *data = POOL_BLOCK_DATA(block); + size_t i, max_pos, alloc_size, used_size; + + used_size = block->size - block->left; + for (i = 0; i < used_size; ) { + alloc_size = *(size_t *)(data + i); + if (alloc_size == 0 || used_size - i < alloc_size) + i_panic("mempool-alloconly: saved alloc size broken"); + i += MEM_ALIGN(sizeof(alloc_size)); + max_pos = i + MEM_ALIGN(alloc_size + SENTRY_COUNT); + i += alloc_size; - for (i = block->size - block->left; i < block->size; i++) { + for (; i < max_pos; i++) { + if (data[i] != CLEAR_CHR) + i_panic("mempool-alloconly: buffer overflow"); + } + } + + if (i != used_size) + i_panic("mempool-alloconly: used_size wrong"); + + /* The unused data must be NULs */ + for (; i < block->size; i++) { if (data[i] != '\0') i_unreached(); } if (block->prev != NULL) - check_nuls(block->prev); + check_sentries(block->prev); } #endif pool_t pool_alloconly_create(const char *name ATTR_UNUSED, size_t size) { struct alloconly_pool apool, *new_apool; - size_t min_alloc = MEM_ALIGN(sizeof(struct alloconly_pool)) + - SIZEOF_POOLBLOCK; + size_t min_alloc = SIZEOF_POOLBLOCK + + MEM_ALIGN(sizeof(struct alloconly_pool) + SENTRY_COUNT); #ifdef DEBUG - min_alloc += MEM_ALIGN(strlen(name) + 1); + min_alloc += MEM_ALIGN(strlen(name) + 1 + SENTRY_COUNT) + + sizeof(size_t)*2; #endif /* create a fake alloconly_pool so we can call block_alloc() */ @@ -115,8 +143,6 @@ /* now allocate the actual alloconly_pool from the created block */ new_apool = p_new(&apool.pool, struct alloconly_pool, 1); *new_apool = apool; - /* the pool allocation must be from the first block */ - i_assert(apool.block->prev == NULL); #ifdef DEBUG if (strncmp(name, MEMPOOL_GROWING, strlen(MEMPOOL_GROWING)) == 0) { name += strlen(MEMPOOL_GROWING); @@ -128,6 +154,8 @@ new_apool->base_size = new_apool->block->size - new_apool->block->left; new_apool->block->last_alloc_size = 0; #endif + /* the first pool allocations must be from the first block */ + i_assert(new_apool->block->prev == NULL); return &new_apool->pool; } @@ -153,10 +181,12 @@ /* destroy the last block */ block = apool->block; #ifdef DEBUG - safe_memset(block, 0xde, SIZEOF_POOLBLOCK + apool->block->size); + safe_memset(block, CLEAR_CHR, SIZEOF_POOLBLOCK + apool->block->size); #else - if (apool->clean_frees) - safe_memset(block, 0, SIZEOF_POOLBLOCK + apool->block->size); + if (apool->clean_frees) { + safe_memset(block, CLEAR_CHR, + SIZEOF_POOLBLOCK + apool->block->size); + } #endif #ifndef USE_GC @@ -238,22 +268,34 @@ { struct alloconly_pool *apool = (struct alloconly_pool *)pool; void *mem; + size_t alloc_size; if (unlikely(size == 0 || size > SSIZE_T_MAX)) i_panic("Trying to allocate %"PRIuSIZE_T" bytes", size); - size = MEM_ALIGN(size); +#ifndef DEBUG + alloc_size = MEM_ALIGN(size); +#else + alloc_size = MEM_ALIGN(sizeof(size)) + MEM_ALIGN(size + SENTRY_COUNT); +#endif - if (apool->block->left < size) { + if (apool->block->left < alloc_size) { /* we need a new block */ - block_alloc(apool, size + SIZEOF_POOLBLOCK); + block_alloc(apool, alloc_size + SIZEOF_POOLBLOCK); } mem = POOL_BLOCK_DATA(apool->block) + (apool->block->size - apool->block->left); - apool->block->left -= size; - apool->block->last_alloc_size = size; + apool->block->left -= alloc_size; + apool->block->last_alloc_size = alloc_size; +#ifdef DEBUG + memcpy(mem, &size, sizeof(size)); + mem = PTR_OFFSET(mem, MEM_ALIGN(sizeof(size))); + /* write CLEAR_CHRs to sentry */ + memset(PTR_OFFSET(mem, size), CLEAR_CHR, + MEM_ALIGN(size + SENTRY_COUNT) - size); +#endif return mem; } @@ -325,7 +367,7 @@ size_t base_size, avail_size; #ifdef DEBUG - check_nuls(apool->block); + check_sentries(apool->block); #endif /* destroy all blocks but the oldest, which contains the @@ -335,10 +377,12 @@ apool->block = block->prev; #ifdef DEBUG - safe_memset(block, 0xde, SIZEOF_POOLBLOCK + block->size); + safe_memset(block, CLEAR_CHR, SIZEOF_POOLBLOCK + block->size); #else - if (apool->clean_frees) - safe_memset(block, 0, SIZEOF_POOLBLOCK + block->size); + if (apool->clean_frees) { + safe_memset(block, CLEAR_CHR, + SIZEOF_POOLBLOCK + block->size); + } #endif #ifndef USE_GC free(block);
--- a/src/lib/module-dir.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/module-dir.c Mon Jun 09 05:11:18 2008 +0300 @@ -250,9 +250,9 @@ module_pos = &modules; for (i = 0; i < count; i++) T_BEGIN { - const char *name = names_p[i]; const char *path, *stripped_name; + name = names_p[i]; stripped_name = module_file_get_name(name); if (!module_want_load(module_names_arr, stripped_name)) module = NULL;
--- a/src/lib/network.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/network.c Mon Jun 09 05:11:18 2008 +0300 @@ -703,3 +703,52 @@ return TRUE; } + +bool net_is_in_network(const struct ip_addr *ip, + const struct ip_addr *net_ip, unsigned int bits) +{ + const uint32_t *ip1, *ip2; + uint32_t mask, i1, i2; + unsigned int pos, i; + + if (IPADDR_IS_V4(ip) != IPADDR_IS_V4(net_ip)) { + /* one is IPv6 and one is IPv4 */ + return FALSE; + } + i_assert(IPADDR_IS_V6(ip) == IPADDR_IS_V6(net_ip)); + + if (IPADDR_IS_V4(ip)) { + ip1 = &ip->u.ip4.s_addr; + ip2 = &net_ip->u.ip4.s_addr; + } else { +#ifdef HAVE_IPV6 + ip1 = (const void *)&ip->u.ip6; + ip2 = (const void *)&net_ip->u.ip6; +#else + /* shouldn't get here */ + return FALSE; +#endif + } + + /* check first the full 32bit ints */ + for (pos = 0, i = 0; pos + 32 <= bits; pos += 32, i++) { + if (ip1[i] != ip2[i]) + return FALSE; + } + i1 = htonl(ip1[i]); + i2 = htonl(ip2[i]); + + /* check the last full bytes */ + for (mask = 0xff000000; pos + 8 <= bits; pos += 8, mask >>= 8) { + if ((i1 & mask) != (i2 & mask)) + return FALSE; + } + + /* check the last bits, they're reversed in bytes */ + bits -= pos; + for (mask = 0x80000000 >> (pos % 32); bits > 0; bits--, mask >>= 1) { + if ((i1 & mask) != (i2 & mask)) + return FALSE; + } + return TRUE; +}
--- a/src/lib/network.h Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/network.h Mon Jun 09 05:11:18 2008 +0300 @@ -110,4 +110,8 @@ bool is_ipv4_address(const char *addr); bool is_ipv6_address(const char *addr); +/* Returns TRUE if ip is in net_ip/bits network. */ +bool net_is_in_network(const struct ip_addr *ip, + const struct ip_addr *net_ip, unsigned int bits); + #endif
--- a/src/lib/randgen.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/randgen.c Mon Jun 09 05:11:18 2008 +0300 @@ -7,6 +7,8 @@ #ifdef HAVE_DEV_URANDOM +#define URANDOM_PATH "/dev/urandom" + #include "fd-close-on-exec.h" #include <unistd.h> #include <fcntl.h> @@ -22,10 +24,16 @@ i_assert(init_refcount > 0); i_assert(size < SSIZE_T_MAX); - for (pos = 0; pos < size; pos += ret) { + for (pos = 0; pos < size; ) { ret = read(urandom_fd, (char *) buf + pos, size - pos); - if (unlikely(ret < 0 && errno != EINTR)) - i_fatal("Error reading from /dev/urandom: %m"); + if (unlikely(ret <= 0)) { + if (ret == 0) + i_fatal("EOF when reading from "URANDOM_PATH); + else if (errno != EINTR) + i_fatal("read("URANDOM_PATH") failed: %m"); + } else { + pos += ret; + } } } @@ -36,13 +44,13 @@ if (init_refcount++ > 0) return; - urandom_fd = open("/dev/urandom", O_RDONLY); + urandom_fd = open(URANDOM_PATH, O_RDONLY); if (urandom_fd == -1) { if (errno == ENOENT) { - i_fatal("/dev/urandom doesn't exist, " + i_fatal(URANDOM_PATH" doesn't exist, " "currently we require it"); } else { - i_fatal("Can't open /dev/urandom: %m"); + i_fatal("Can't open "URANDOM_PATH": %m"); } }
--- a/src/lib/restrict-access.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/restrict-access.c Mon Jun 09 05:11:18 2008 +0300 @@ -12,8 +12,9 @@ #include <time.h> #include <grp.h> -static gid_t primary_gid = (gid_t)-1, privileged_gid = (gid_t)-1; -static bool using_priv_gid = FALSE; +static gid_t process_primary_gid = (gid_t)-1; +static gid_t process_privileged_gid = (gid_t)-1; +static bool process_using_priv_gid = FALSE; void restrict_access_set_env(const char *user, uid_t uid, gid_t gid, gid_t privileged_gid, @@ -160,7 +161,7 @@ /* if we're using a privileged GID, we can temporarily drop our effective GID. we still want to be able to use its privileges, so add it to supplementary groups. */ - add_primary_gid = privileged_gid != (gid_t)-1; + add_primary_gid = process_privileged_gid != (gid_t)-1; tmp = extra_groups == NULL ? &empty : t_strsplit_spaces(extra_groups, ", "); @@ -171,7 +172,7 @@ have_root_group); /* see if the list already contains the primary GID */ for (i = 0; i < gid_count; i++) { - if (gid_list[i] == primary_gid) { + if (gid_list[i] == process_primary_gid) { add_primary_gid = FALSE; break; } @@ -184,7 +185,7 @@ /* Some OSes don't like an empty groups list, so use the primary GID as the only one. */ gid_list = t_new(gid_t, 2); - gid_list[0] = primary_gid; + gid_list[0] = process_primary_gid; gid_count = 1; add_primary_gid = FALSE; } @@ -195,11 +196,11 @@ memcpy(gid_list2, gid_list, gid_count * sizeof(gid_t)); for (; *tmp != NULL; tmp++) { gid = get_group_id(*tmp); - if (gid != primary_gid) + if (gid != process_primary_gid) gid_list2[gid_count++] = gid; } if (add_primary_gid) - gid_list2[gid_count++] = primary_gid; + gid_list2[gid_count++] = process_primary_gid; gid_list = gid_list2; } @@ -224,28 +225,30 @@ /* set the primary/privileged group */ env = getenv("RESTRICT_SETGID"); - primary_gid = env == NULL || *env == '\0' ? (gid_t)-1 : + process_primary_gid = env == NULL || *env == '\0' ? (gid_t)-1 : (gid_t)strtoul(env, NULL, 10); env = getenv("RESTRICT_SETGID_PRIV"); - privileged_gid = env == NULL || *env == '\0' ? (gid_t)-1 : + process_privileged_gid = env == NULL || *env == '\0' ? (gid_t)-1 : (gid_t)strtoul(env, NULL, 10); - have_root_group = primary_gid == 0; - if (primary_gid != (gid_t)-1 || privileged_gid != (gid_t)-1) { - if (primary_gid == (gid_t)-1) - primary_gid = getegid(); - restrict_init_groups(primary_gid, privileged_gid); + have_root_group = process_primary_gid == 0; + if (process_primary_gid != (gid_t)-1 || + process_privileged_gid != (gid_t)-1) { + if (process_primary_gid == (gid_t)-1) + process_primary_gid = getegid(); + restrict_init_groups(process_primary_gid, + process_privileged_gid); } else { - if (primary_gid == (gid_t)-1) - primary_gid = getegid(); + if (process_primary_gid == (gid_t)-1) + process_primary_gid = getegid(); } /* set system user's groups */ env = getenv("RESTRICT_USER"); if (env != NULL && *env != '\0' && is_root) { - if (initgroups(env, primary_gid) < 0) { + if (initgroups(env, process_primary_gid) < 0) { i_fatal("initgroups(%s, %s) failed: %m", - env, dec2str(primary_gid)); + env, dec2str(process_primary_gid)); } preserve_groups = TRUE; } @@ -303,18 +306,18 @@ env = getenv("RESTRICT_GID_FIRST"); if (env != NULL && atoi(env) != 0) allow_root_gid = FALSE; - else if (primary_gid == 0 || privileged_gid == 0) + else if (process_primary_gid == 0 || process_privileged_gid == 0) allow_root_gid = TRUE; else allow_root_gid = FALSE; if (!allow_root_gid && uid != 0) { if (getgid() == 0 || getegid() == 0 || setgid(0) == 0) { - if (primary_gid == 0) + if (process_primary_gid == 0) i_fatal("GID 0 isn't permitted"); i_fatal("We couldn't drop root group privileges " "(wanted=%s, gid=%s, egid=%s)", - dec2str(primary_gid), + dec2str(process_primary_gid), dec2str(getgid()), dec2str(getegid())); } } @@ -323,7 +326,7 @@ env_put("RESTRICT_USER="); env_put("RESTRICT_CHROOT="); env_put("RESTRICT_SETUID="); - if (privileged_gid == (gid_t)-1) { + if (process_privileged_gid == (gid_t)-1) { /* if we're dropping privileges before executing and a privileged group is set, the groups must be fixed after exec */ @@ -337,29 +340,29 @@ int restrict_access_use_priv_gid(void) { - i_assert(!using_priv_gid); + i_assert(!process_using_priv_gid); - if (privileged_gid == (gid_t)-1) + if (process_privileged_gid == (gid_t)-1) return 0; - if (setegid(privileged_gid) < 0) { + if (setegid(process_privileged_gid) < 0) { i_error("setegid(privileged) failed: %m"); return -1; } - using_priv_gid = TRUE; + process_using_priv_gid = TRUE; return 0; } void restrict_access_drop_priv_gid(void) { - if (!using_priv_gid) + if (!process_using_priv_gid) return; - if (setegid(primary_gid) < 0) + if (setegid(process_primary_gid) < 0) i_fatal("setegid(primary) failed: %m"); - using_priv_gid = FALSE; + process_using_priv_gid = FALSE; } bool restrict_access_have_priv_gid(void) { - return privileged_gid != (gid_t)-1; + return process_privileged_gid != (gid_t)-1; }
--- a/src/lib/str-find.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/str-find.c Mon Jun 09 05:11:18 2008 +0300 @@ -1,5 +1,7 @@ /* Copyright (c) 2007-2008 Dovecot authors, see the included COPYING file */ +/* @UNSAFE: whole file */ + #include "lib.h" #include "str-find.h" @@ -53,7 +55,7 @@ unsigned int j, *suffixes; int i; - suffixes = t_buffer_get(ctx->key_len); + suffixes = t_buffer_get(sizeof(*suffixes) * ctx->key_len); init_suffixes(ctx, suffixes); for (i = 0; i < (int)ctx->key_len; i++) @@ -71,7 +73,7 @@ for (i = 0; i <= (int)ctx->key_len - 2; i++) ctx->goodtab[len_1 - suffixes[i]] = len_1 - i; } - + struct str_find_context *str_find_init(pool_t pool, const char *key) { struct str_find_context *ctx; @@ -80,7 +82,7 @@ ctx = p_malloc(pool, sizeof(struct str_find_context) + sizeof(ctx->goodtab[0]) * key_len); ctx->pool = pool; - ctx->matches = p_malloc(pool, key_len); + ctx->matches = p_new(pool, unsigned int, key_len); ctx->key_len = key_len; ctx->key = p_malloc(pool, key_len); memcpy(ctx->key, key, key_len);
--- a/src/lib/var-expand.c Sat May 17 17:50:54 2008 +0300 +++ b/src/lib/var-expand.c Mon Jun 09 05:11:18 2008 +0300 @@ -10,6 +10,7 @@ #include "var-expand.h" #include <stdlib.h> +#include <ctype.h> struct var_expand_context { int offset; @@ -81,7 +82,8 @@ return str_c(hash); } -static const char *m_str_md5(const char *str, struct var_expand_context *ctx ATTR_UNUSED) +static const char * +m_str_md5(const char *str, struct var_expand_context *ctx ATTR_UNUSED) { unsigned char digest[16]; @@ -90,7 +92,8 @@ return binary_to_hex(digest, sizeof(digest)); } -static const char *m_str_ldap_dn(const char *str, struct var_expand_context *ctx ATTR_UNUSED) +static const char * +m_str_ldap_dn(const char *str, struct var_expand_context *ctx ATTR_UNUSED) { string_t *ret = t_str_new(256); @@ -105,6 +108,17 @@ return str_free_without_data(&ret); } +static const char * +m_str_trim(const char *str, struct var_expand_context *ctx ATTR_UNUSED) +{ + unsigned int len; + + len = strlen(str); + while (len > 0 && i_isspace(str[len-1])) + len--; + return t_strndup(str, len); +} + #define MAX_MODIFIER_COUNT 10 static const struct var_expand_modifier modifiers[] = { { 'L', m_str_lcase }, @@ -115,6 +129,7 @@ { 'H', m_str_hash }, { 'M', m_str_md5 }, { 'D', m_str_ldap_dn }, + { 'T', m_str_trim }, { '\0', NULL } };
--- a/src/login-common/ssl-proxy-openssl.c Sat May 17 17:50:54 2008 +0300 +++ b/src/login-common/ssl-proxy-openssl.c Mon Jun 09 05:11:18 2008 +0300 @@ -309,8 +309,11 @@ size_t err_size = 256; err = ERR_get_error(); - if (err == 0) - return strerror(errno); + if (err == 0) { + if (errno != 0) + return strerror(errno); + return "Unknown error"; + } buf = t_malloc(err_size); buf[err_size-1] = '\0'; @@ -806,6 +809,8 @@ ssl_free_parameters(&ssl_params); SSL_CTX_free(ssl_ctx); + EVP_cleanup(); + ERR_free_strings(); } #endif
--- a/src/master/auth-process.c Sat May 17 17:50:54 2008 +0300 +++ b/src/master/auth-process.c Mon Jun 09 05:11:18 2008 +0300 @@ -342,6 +342,7 @@ if (!p->initialized && io_loop_is_running(ioloop) && !p->external) { /* log the process exit and kill ourself */ child_processes_deinit(); + log_deinit(); i_fatal("Auth process died too early - shutting down"); }
--- a/src/master/dict-process.c Sat May 17 17:50:54 2008 +0300 +++ b/src/master/dict-process.c Mon Jun 09 05:11:18 2008 +0300 @@ -25,7 +25,7 @@ struct io *io; }; -static struct dict_process *process; +static struct dict_process *dict_process; static void dict_process_unlisten(struct dict_process *process); @@ -84,6 +84,12 @@ child_process_init_env(); env_put(t_strconcat("DICT_LISTEN_FROM_FD=", process->path, NULL)); + if (settings_root->defaults->dict_db_config != NULL) { + env_put(t_strconcat("DB_CONFIG=", + settings_root->defaults->dict_db_config, + NULL)); + } + dicts = array_get(&settings_root->dicts, &count); i_assert((count % 2) == 0); for (i = 0; i < count; i += 2) @@ -175,7 +181,9 @@ void dict_process_init(void) { - process = i_new(struct dict_process, 1); + struct dict_process *process; + + process = dict_process = i_new(struct dict_process, 1); process->process.type = PROCESS_TYPE_DICT; process->fd = -1; process->path = i_strconcat(settings_root->defaults->base_dir, @@ -188,6 +196,8 @@ void dict_process_deinit(void) { + struct dict_process *process = dict_process; + dict_process_unlisten(process); if (process->log != NULL) log_unref(process->log); @@ -197,6 +207,8 @@ void dict_process_kill(void) { + struct dict_process *process = dict_process; + if (process->log != NULL) { log_unref(process->log); process->log = NULL;
--- a/src/master/listener.c Sat May 17 17:50:54 2008 +0300 +++ b/src/master/listener.c Mon Jun 09 05:11:18 2008 +0300 @@ -130,7 +130,7 @@ } static void -listener_init(const char *set_name, const char *listen, +listener_init(const char *set_name, const char *listen_list, unsigned int default_port, ARRAY_TYPE(listener) *listens_arr) { const char *const *tmp; @@ -148,7 +148,7 @@ l.fd = -1; l.wanted = TRUE; - for (tmp = t_strsplit_spaces(listen, ", "); *tmp != NULL; tmp++) { + for (tmp = t_strsplit_spaces(listen_list, ", "); *tmp != NULL; tmp++) { l.port = default_port; resolve_ip(set_name, *tmp, &l.ip, &l.port); @@ -204,7 +204,7 @@ { const char *const *proto; unsigned int default_port; - bool listen = FALSE, ssl_listen = FALSE; + bool nonssl_listen = FALSE, ssl_listen = FALSE; if (set == NULL) return; @@ -214,14 +214,14 @@ for (; *proto != NULL; proto++) { if (strcasecmp(*proto, "imap") == 0) { if (set->protocol == MAIL_PROTOCOL_IMAP) - listen = TRUE; + nonssl_listen = TRUE; } else if (strcasecmp(*proto, "imaps") == 0) { if (set->protocol == MAIL_PROTOCOL_IMAP && !set->ssl_disable) ssl_listen = TRUE; } else if (strcasecmp(*proto, "pop3") == 0) { if (set->protocol == MAIL_PROTOCOL_POP3) - listen = TRUE; + nonssl_listen = TRUE; } else if (strcasecmp(*proto, "pop3s") == 0) { if (set->protocol == MAIL_PROTOCOL_POP3 && !set->ssl_disable) @@ -229,7 +229,7 @@ } } - if (!listen) + if (!nonssl_listen) listener_close_fds(&set->listens); else { default_port = set->protocol == MAIL_PROTOCOL_IMAP ? 143 : 110;
--- a/src/master/login-process.c Sat May 17 17:50:54 2008 +0300 +++ b/src/master/login-process.c Mon Jun 09 05:11:18 2008 +0300 @@ -689,7 +689,10 @@ fd_limit = 16 + listen_count + ssl_listen_count + 2 * (group->set->login_process_per_connection ? 1 : group->set->login_max_connections); - restrict_fd_limit(fd_limit); +#ifdef DEBUG + if (!gdb) +#endif + restrict_fd_limit(fd_limit); /* make sure we don't leak syslog fd, but do it last so that any errors above will be logged */
--- a/src/master/mail-process.c Sat May 17 17:50:54 2008 +0300 +++ b/src/master/mail-process.c Mon Jun 09 05:11:18 2008 +0300 @@ -519,7 +519,7 @@ enum master_login_status create_mail_process(enum process_type process_type, struct settings *set, - int socket, const struct ip_addr *local_ip, + int socket_fd, const struct ip_addr *local_ip, const struct ip_addr *remote_ip, const char *user, const char *const *args, bool dump_capability) @@ -535,8 +535,8 @@ uid_t uid; gid_t gid; ARRAY_DEFINE(extra_args, const char *); - unsigned int i, count, left, process_count, throttle; - int ret, log_fd, nice, chdir_errno; + unsigned int i, len, count, left, process_count, throttle; + int ret, log_fd, nice_value, chdir_errno; bool home_given, nfs_check; i_assert(process_type == PROCESS_TYPE_IMAP || @@ -558,7 +558,7 @@ t_array_init(&extra_args, 16); mail = home_dir = chroot_dir = system_user = ""; - uid = (uid_t)-1; gid = (gid_t)-1; nice = 0; + uid = (uid_t)-1; gid = (gid_t)-1; nice_value = 0; home_given = FALSE; for (; *args != NULL; args++) { if (strncmp(*args, "home=", 5) == 0) { @@ -569,7 +569,7 @@ else if (strncmp(*args, "chroot=", 7) == 0) chroot_dir = *args + 7; else if (strncmp(*args, "nice=", 5) == 0) - nice = atoi(*args + 5); + nice_value = atoi(*args + 5); else if (strncmp(*args, "system_user=", 12) == 0) system_user = *args + 12; else if (strncmp(*args, "uid=", 4) == 0) { @@ -638,6 +638,12 @@ chroot_dir, user); return MASTER_LOGIN_STATUS_INTERNAL_ERROR; } + len = strlen(chroot_dir); + if (len > 2 && strcmp(chroot_dir + len - 2, "/.") == 0 && + strncmp(home_dir, chroot_dir, len - 2) == 0) { + /* strip chroot dir from home dir */ + home_dir += len - 2; + } if (!dump_capability) { throttle = set->mail_debug ? 0 : @@ -699,9 +705,9 @@ } #ifdef HAVE_SETPRIORITY - if (nice != 0) { - if (setpriority(PRIO_PROCESS, 0, nice) < 0) - i_error("setpriority(%d) failed: %m", nice); + if (nice_value != 0) { + if (setpriority(PRIO_PROCESS, 0, nice_value) < 0) + i_error("setpriority(%d) failed: %m", nice_value); } #endif @@ -714,9 +720,9 @@ child_process_init_env(); /* move the client socket into stdin and stdout fds, log to stderr */ - if (dup2(dump_capability ? null_fd : socket, 0) < 0) + if (dup2(dump_capability ? null_fd : socket_fd, 0) < 0) i_fatal("dup2(stdin) failed: %m"); - if (dup2(socket, 1) < 0) + if (dup2(socket_fd, 1) < 0) i_fatal("dup2(stdout) failed: %m"); if (dup2(log_fd, 2) < 0) i_fatal("dup2(stderr) failed: %m"); @@ -736,7 +742,7 @@ if (dump_capability) env_put("DUMP_CAPABILITY=1"); - if (*home_dir == '\0') { + if (*home_dir == '\0' && *chroot_dir == '\0') { full_home_dir = ""; ret = -1; } else {
--- a/src/master/mail-process.h Sat May 17 17:50:54 2008 +0300 +++ b/src/master/mail-process.h Mon Jun 09 05:11:18 2008 +0300 @@ -10,7 +10,7 @@ enum master_login_status create_mail_process(enum process_type process_type, struct settings *set, - int socket, const struct ip_addr *local_ip, + int socket_fd, const struct ip_addr *local_ip, const struct ip_addr *remote_ip, const char *user, const char *const *args, bool dump_capability);
--- a/src/master/master-settings-defs.c Sat May 17 17:50:54 2008 +0300 +++ b/src/master/master-settings-defs.c Mon Jun 09 05:11:18 2008 +0300 @@ -124,5 +124,8 @@ DEF_STR(pop3_client_workarounds), DEF_STR(pop3_logout_format), + /* dict */ + DEF_STR(dict_db_config), + { 0, NULL, 0 } };
--- a/src/master/master-settings.c Sat May 17 17:50:54 2008 +0300 +++ b/src/master/master-settings.c Mon Jun 09 05:11:18 2008 +0300 @@ -290,6 +290,9 @@ MEMBER(pop3_client_workarounds) "", MEMBER(pop3_logout_format) "top=%t/%p, retr=%r/%b, del=%d/%m, size=%s", + /* dict */ + MEMBER(dict_db_config) NULL, + /* .. */ }; @@ -1647,7 +1650,7 @@ { const struct auth_passdb_settings *passdb; const struct auth_userdb_settings *userdb; - const struct auth_socket_settings *socket; + const struct auth_socket_settings *socket_set; const void *sets[2], *sets2[2]; const void *empty_defaults; @@ -1679,23 +1682,23 @@ nondefaults, 4); } - socket = auth->sockets; - for (; socket != NULL; socket = socket->next) { + socket_set = auth->sockets; + for (; socket_set != NULL; socket_set = socket_set->next) { printf(" socket:\n"); - sets2[1] = socket; + sets2[1] = socket_set; settings_dump(auth_socket_setting_defs, sets2, NULL, 2, nondefaults, 4); - if (socket->client.used) { + if (socket_set->client.used) { printf(" client:\n"); - sets2[1] = &socket->client; + sets2[1] = &socket_set->client; settings_dump(socket_setting_defs, sets2, NULL, 2, nondefaults, 6); } - if (socket->master.used) { + if (socket_set->master.used) { printf(" master:\n"); - sets2[1] = &socket->master; + sets2[1] = &socket_set->master; settings_dump(socket_setting_defs, sets2, NULL, 2, nondefaults, 6); }
--- a/src/master/master-settings.h Sat May 17 17:50:54 2008 +0300 +++ b/src/master/master-settings.h Mon Jun 09 05:11:18 2008 +0300 @@ -136,6 +136,9 @@ const char *pop3_client_workarounds; const char *pop3_logout_format; + /* dict */ + const char *dict_db_config; + /* .. */ ARRAY_TYPE(listener) listens; ARRAY_TYPE(listener) ssl_listens;
--- a/src/plugins/acl/acl-backend-vfile-acllist.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/acl/acl-backend-vfile-acllist.c Mon Jun 09 05:11:18 2008 +0300 @@ -203,7 +203,12 @@ the file at the same time the result should be the same. */ fd = safe_mkstemp(path, mode, (uid_t)-1, gid); if (fd == -1) { - i_error("safe_mkstemp(%s) failed: %m", str_c(path)); + if (errno == EACCES) { + /* Ignore silently if we can't create it */ + return 0; + } + i_error("dovecot-acl-list creation failed: " + "safe_mkstemp(%s) failed: %m", str_c(path)); return -1; } output = o_stream_create_fd_file(fd, 0, FALSE);
--- a/src/plugins/acl/acl-backend-vfile.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/acl/acl-backend-vfile.c Mon Jun 09 05:11:18 2008 +0300 @@ -94,9 +94,16 @@ return 0; } -static void acl_backend_vfile_deinit(struct acl_backend *backend) +static void acl_backend_vfile_deinit(struct acl_backend *_backend) { - pool_unref(&backend->pool); + struct acl_backend_vfile *backend = + (struct acl_backend_vfile *)_backend; + + if (backend->acllist_pool != NULL) { + array_free(&backend->acllist); + pool_unref(&backend->acllist_pool); + } + pool_unref(&backend->backend.pool); } static struct acl_object *
--- a/src/plugins/acl/acl-cache.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/acl/acl-cache.c Mon Jun 09 05:11:18 2008 +0300 @@ -62,6 +62,8 @@ struct acl_cache *cache = *_cache; *_cache = NULL; + + acl_cache_flush_all(cache); array_free(&cache->right_idx_name_map); hash_destroy(&cache->right_name_idx_map); hash_destroy(&cache->objects);
--- a/src/plugins/acl/acl-cache.h Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/acl/acl-cache.h Mon Jun 09 05:11:18 2008 +0300 @@ -13,9 +13,8 @@ /* variable length bitmask */ unsigned char mask[1]; }; -#define SIZEOF_ACL_MASK(count) \ - (sizeof(pool_t) + sizeof(unsigned int) + \ - (count + CHAR_BIT-1) / CHAR_BIT) +#define SIZEOF_ACL_MASK(bitmask_size) \ + (sizeof(pool_t) + sizeof(unsigned int) + (bitmask_size)) struct acl_cache *acl_cache_init(struct acl_backend *backend, size_t validity_rec_size);
--- a/src/plugins/acl/acl-mailbox-list.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/acl/acl-mailbox-list.c Mon Jun 09 05:11:18 2008 +0300 @@ -29,7 +29,6 @@ struct mailbox_list_iterate_context *super_ctx; struct mailbox_tree_context *tree; - struct mailbox_tree_iterate_context *tree_iter; struct mailbox_info info; }; @@ -63,7 +62,7 @@ can_see_r); } -static bool +static void acl_mailbox_try_list_fast(struct acl_mailbox_list_iterate_context *ctx, const char *const *patterns) { @@ -73,19 +72,20 @@ alist->rights.acl_storage_right_idx + ACL_STORAGE_RIGHT_LOOKUP; const struct acl_mask *acl_mask; struct acl_mailbox_list_context *nonowner_list_ctx; - struct imap_match_glob *glob; struct mail_namespace *ns = ctx->ctx.list->ns; + struct mailbox_list_iter_update_context update_ctx; const char *name; string_t *vname; char sep; int try, ret; - if ((ctx->ctx.flags & MAILBOX_LIST_ITER_RAW_LIST) != 0) - return FALSE; + if ((ctx->ctx.flags & (MAILBOX_LIST_ITER_RAW_LIST | + MAILBOX_LIST_ITER_SELECT_SUBSCRIBED)) != 0) + return; if (acl_backend_get_default_rights(backend, &acl_mask) < 0 || acl_cache_mask_isset(acl_mask, *idxp)) - return FALSE; + return; /* default is to not list mailboxes. we can optimize this. */ if ((ctx->ctx.flags & MAILBOX_LIST_ITER_VIRTUAL_NAMES) != 0) { @@ -95,13 +95,19 @@ sep = ns->real_sep; vname = NULL; } - glob = imap_match_init_multiple(pool_datastack_create(), patterns, - TRUE, sep); + + memset(&update_ctx, 0, sizeof(update_ctx)); + update_ctx.iter_ctx = &ctx->ctx; + update_ctx.glob = + imap_match_init_multiple(pool_datastack_create(), patterns, + TRUE, sep);; + update_ctx.match_parents = TRUE; for (try = 0; try < 2; try++) { nonowner_list_ctx = acl_backend_nonowner_lookups_iter_init(backend); ctx->tree = mailbox_tree_init(sep); + update_ctx.tree_ctx = ctx->tree; while ((ret = acl_backend_nonowner_lookups_iter_next( nonowner_list_ctx, &name)) > 0) { @@ -109,22 +115,16 @@ name = mail_namespace_get_vname(ns, vname, name); } - mailbox_list_iter_update(&ctx->ctx, ctx->tree, - glob, FALSE, TRUE, name); + mailbox_list_iter_update(&update_ctx, name); } + acl_backend_nonowner_lookups_iter_deinit(&nonowner_list_ctx); + if (ret == 0) break; /* try again */ mailbox_tree_deinit(&ctx->tree); - acl_backend_nonowner_lookups_iter_deinit(&nonowner_list_ctx); } - if (ret < 0) - return FALSE; - - ctx->tree_iter = mailbox_tree_iterate_init(ctx->tree, NULL, - MAILBOX_FLAG_MATCHED); - return TRUE; } static struct mailbox_list_iterate_context * @@ -134,19 +134,16 @@ { struct acl_mailbox_list *alist = ACL_LIST_CONTEXT(list); struct acl_mailbox_list_iterate_context *ctx; - bool ret; ctx = i_new(struct acl_mailbox_list_iterate_context, 1); ctx->ctx.list = list; ctx->ctx.flags = flags; T_BEGIN { - ret = acl_mailbox_try_list_fast(ctx, patterns); + acl_mailbox_try_list_fast(ctx, patterns); } T_END; - if (!ret) { - ctx->super_ctx = alist->module_ctx.super. - iter_init(list, patterns, flags); - } + ctx->super_ctx = alist->module_ctx.super. + iter_init(list, patterns, flags); return &ctx->ctx; } @@ -154,16 +151,34 @@ acl_mailbox_list_iter_next_info(struct acl_mailbox_list_iterate_context *ctx) { struct acl_mailbox_list *alist = ACL_LIST_CONTEXT(ctx->ctx.list); - struct mailbox_node *node; + const struct mailbox_info *info; - if (ctx->tree_iter == NULL) - return alist->module_ctx.super.iter_next(ctx->super_ctx); + do { + info = alist->module_ctx.super.iter_next(ctx->super_ctx); + if (info == NULL) + return NULL; + /* if the mailbox isn't in shared mailboxes list, it's not + visible to us. */ + } while (ctx->tree != NULL && + mailbox_tree_lookup(ctx->tree, info->name) == NULL); + + return info; +} - node = mailbox_tree_iterate_next(ctx->tree_iter, &ctx->info.name); - if (node == NULL) - return NULL; - ctx->info.flags = node->flags; - return &ctx->info; +static const char * +acl_mailbox_list_iter_get_name(struct mailbox_list_iterate_context *ctx, + const char *name) +{ + struct mail_namespace *ns = ctx->list->ns; + + if ((ctx->flags & MAILBOX_LIST_ITER_VIRTUAL_NAMES) == 0) + return name; + + /* Mailbox names contain namespace prefix, + except when listing INBOX. */ + if (strncmp(name, ns->prefix, ns->prefix_len) == 0) + name += ns->prefix_len; + return mail_namespace_fix_sep(ns, name); } static int @@ -171,7 +186,6 @@ const struct mailbox_info *info) { struct acl_mailbox_list *alist = ACL_LIST_CONTEXT(ctx->ctx.list); - struct mail_namespace *ns = ctx->ctx.list->ns; const char *acl_name; int ret; @@ -180,15 +194,7 @@ return 1; } - acl_name = info->name; - if ((ctx->ctx.flags & MAILBOX_LIST_ITER_VIRTUAL_NAMES) != 0) { - /* Mailbox names contain namespace prefix, - except when listing INBOX. */ - if (strncmp(acl_name, ns->prefix, ns->prefix_len) == 0) - acl_name += ns->prefix_len; - acl_name = mail_namespace_fix_sep(ns, acl_name); - } - + acl_name = acl_mailbox_list_iter_get_name(&ctx->ctx, info->name); ret = acl_mailbox_list_have_right(alist, acl_name, ACL_STORAGE_RIGHT_LOOKUP, NULL); @@ -233,6 +239,27 @@ } static int +acl_mailbox_list_iter_is_mailbox(struct mailbox_list_iterate_context *ctx, + const char *dir, const char *fname, + const char *mailbox_name, + enum mailbox_list_file_type type, + enum mailbox_info_flags *flags_r) +{ + struct acl_mailbox_list *alist = ACL_LIST_CONTEXT(ctx->list); + int ret; + + ret = alist->module_ctx.super.iter_is_mailbox(ctx, dir, fname, + mailbox_name, + type, flags_r); + if (ret <= 0 || (ctx->flags & MAILBOX_LIST_ITER_RAW_LIST) != 0) + return ret; + + mailbox_name = acl_mailbox_list_iter_get_name(ctx, mailbox_name); + return acl_mailbox_list_have_right(alist, mailbox_name, + ACL_STORAGE_RIGHT_LOOKUP, NULL); +} + +static int acl_mailbox_list_iter_deinit(struct mailbox_list_iterate_context *_ctx) { struct acl_mailbox_list_iterate_context *ctx = @@ -240,12 +267,10 @@ struct acl_mailbox_list *alist = ACL_LIST_CONTEXT(_ctx->list); int ret = ctx->ctx.failed ? -1 : 0; - if (ctx->super_ctx != NULL) { - if (alist->module_ctx.super.iter_deinit(ctx->super_ctx) < 0) - ret = -1; - } - if (ctx->tree_iter != NULL) - mailbox_tree_iterate_deinit(&ctx->tree_iter); + if (alist->module_ctx.super.iter_deinit(ctx->super_ctx) < 0) + ret = -1; + if (ctx->tree != NULL) + mailbox_tree_deinit(&ctx->tree); i_free(ctx); return ret; @@ -387,9 +412,6 @@ const char *acl_env, *current_username, *owner_username; bool owner = TRUE; - if (acl_next_hook_mailbox_list_created != NULL) - acl_next_hook_mailbox_list_created(list); - acl_env = getenv("ACL"); i_assert(acl_env != NULL); @@ -430,6 +452,7 @@ list->v.iter_init = acl_mailbox_list_iter_init; list->v.iter_next = acl_mailbox_list_iter_next; list->v.iter_deinit = acl_mailbox_list_iter_deinit; + list->v.iter_is_mailbox = acl_mailbox_list_iter_is_mailbox; list->v.get_mailbox_name_status = acl_get_mailbox_name_status; list->v.delete_mailbox = acl_mailbox_list_delete; list->v.rename_mailbox = acl_mailbox_list_rename; @@ -437,4 +460,7 @@ acl_storage_rights_ctx_init(&alist->rights, backend); MODULE_CONTEXT_SET(list, acl_mailbox_list_module, alist); + + if (acl_next_hook_mailbox_list_created != NULL) + acl_next_hook_mailbox_list_created(list); }
--- a/src/plugins/acl/acl-mailbox.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/acl/acl-mailbox.c Mon Jun 09 05:11:18 2008 +0300 @@ -155,7 +155,9 @@ /* handle this by first removing the allowed flags and then adding the allowed flags */ acl_mail_update_flags(_mail, MODIFY_REMOVE, ~flags); - acl_mail_update_flags(_mail, MODIFY_ADD, flags); + if (flags != 0) + acl_mail_update_flags(_mail, MODIFY_ADD, flags); + return; } amail->super.update_flags(_mail, modify_type, flags); @@ -295,6 +297,36 @@ transaction_commit(ctx, uid_validity_r, first_saved_uid_r, last_saved_uid_r); } + +static int +acl_keywords_create(struct mailbox *box, const char *const keywords[], + struct mail_keywords **keywords_r, bool skip_invalid) +{ + struct acl_mailbox *abox = ACL_CONTEXT(box); + int ret; + + ret = mailbox_acl_right_lookup(box, ACL_STORAGE_RIGHT_WRITE); + if (ret < 0) { + if (!skip_invalid) + return -1; + /* we can't return failure. assume we don't have permissions. */ + ret = 0; + } + + if (ret == 0) { + /* no permission to update any flags. just return empty + keywords list. */ + const char *null = NULL; + + return abox->module_ctx.super.keywords_create(box, &null, + keywords_r, + skip_invalid); + } + + return abox->module_ctx.super.keywords_create(box, keywords, + keywords_r, skip_invalid); +} + struct mailbox *acl_mailbox_open_box(struct mailbox *box) { struct acl_mail_storage *astorage = ACL_CONTEXT(box->storage); @@ -311,6 +343,7 @@ box->v.close = acl_mailbox_close; box->v.mail_alloc = acl_mail_alloc; box->v.save_init = acl_save_init; + box->v.keywords_create = acl_keywords_create; box->v.copy = acl_copy; box->v.transaction_commit = acl_transaction_commit; MODULE_CONTEXT_SET(box, acl_storage_module, abox);
--- a/src/plugins/acl/acl-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/acl/acl-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -160,9 +160,6 @@ struct acl_mail_storage *astorage; struct acl_backend *backend; - if (acl_next_hook_mail_storage_created != NULL) - acl_next_hook_mail_storage_created(storage); - astorage = p_new(storage->pool, struct acl_mail_storage, 1); astorage->module_ctx.super = storage->v; storage->v.destroy = acl_storage_destroy; @@ -173,5 +170,8 @@ acl_storage_rights_ctx_init(&astorage->rights, backend); MODULE_CONTEXT_SET(storage, acl_storage_module, astorage); + + if (acl_next_hook_mail_storage_created != NULL) + acl_next_hook_mail_storage_created(storage); }
--- a/src/plugins/convert/convert-plugin.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/convert/convert-plugin.c Mon Jun 09 05:11:18 2008 +0300 @@ -12,19 +12,12 @@ static void (*convert_next_hook_mail_namespaces_created) (struct mail_namespace *namespaces); -static void -convert_hook_mail_namespaces_created(struct mail_namespace *namespaces) +static void convert_mail_storage(struct mail_namespace *namespaces, + const char *convert_mail) { - const char *convert_mail, *str; + const char *str; struct convert_settings set; - if (convert_next_hook_mail_namespaces_created != NULL) - convert_next_hook_mail_namespaces_created(namespaces); - - convert_mail = getenv("CONVERT_MAIL"); - if (convert_mail == NULL) - return; - memset(&set, 0, sizeof(set)); set.user = getenv("USER"); if (set.user == NULL) @@ -44,6 +37,19 @@ i_fatal("Mailbox conversion failed, exiting"); } +static void +convert_hook_mail_namespaces_created(struct mail_namespace *namespaces) +{ + const char *convert_mail; + + convert_mail = getenv("CONVERT_MAIL"); + if (convert_mail != NULL) + convert_mail_storage(namespaces, convert_mail); + + if (convert_next_hook_mail_namespaces_created != NULL) + convert_next_hook_mail_namespaces_created(namespaces); +} + void convert_plugin_init(void) { convert_next_hook_mail_namespaces_created =
--- a/src/plugins/convert/convert-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/convert/convert-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -148,6 +148,15 @@ bool t; int ret; + /* create as non-selectable mailbox so the dbox-Mails directory + isn't created yet */ + if (mail_storage_mailbox_create(dest_storage, dest_name, TRUE) < 0) { + i_error("Mailbox conversion: " + "Couldn't create mailbox %s: %s", + dest_name, storage_error(dest_storage)); + return -1; + } + src_path = mail_storage_get_mailbox_path(src_storage, src_name, &t); dest_path = mail_storage_get_mailbox_path(dest_storage, dest_name, &t);
--- a/src/plugins/expire/auth-client.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/expire/auth-client.c Mon Jun 09 05:11:18 2008 +0300 @@ -130,20 +130,24 @@ return; } + if (uid != conn->current_uid && conn->current_uid != 0) { + if (seteuid(0) != 0) + i_fatal("seteuid(0) failed: %m"); + conn->current_uid = 0; + } + + /* change GID */ + restrict_access_by_env(FALSE); + /* we'll change only effective UID. This is a bit unfortunate since it allows reverting back to root, but we'll have to be able to access different users' mailboxes.. */ if (uid != conn->current_uid) { - if (conn->current_uid != 0) { - if (seteuid(0) != 0) - i_fatal("seteuid(0) failed: %m"); - } if (seteuid(uid) < 0) i_fatal("seteuid(%s) failed: %m", dec2str(uid)); conn->current_uid = uid; } - restrict_access_by_env(FALSE); conn->return_value = 1; }
--- a/src/plugins/expire/expire-env.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/expire/expire-env.c Mon Jun 09 05:11:18 2008 +0300 @@ -105,10 +105,17 @@ return expunge_min > 0 || altmove_min > 0; } -unsigned int expire_box_find_min_secs(struct expire_env *env, const char *name) +unsigned int expire_box_find_min_secs(struct expire_env *env, const char *name, + bool *altmove_r) { unsigned int secs1, secs2; (void)expire_box_find(env, name, &secs1, &secs2); - return secs1 < secs2 && secs1 != 0 ? secs1 : secs2; + if (secs1 != 0 && (secs1 < secs2 || secs2 == 0)) { + *altmove_r = FALSE; + return secs1; + } else { + *altmove_r = TRUE; + return secs2; + } }
--- a/src/plugins/expire/expire-env.h Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/expire/expire-env.h Mon Jun 09 05:11:18 2008 +0300 @@ -10,6 +10,7 @@ unsigned int *expunge_secs_r, unsigned int *altmove_secs_r); -unsigned int expire_box_find_min_secs(struct expire_env *env, const char *name); +unsigned int expire_box_find_min_secs(struct expire_env *env, const char *name, + bool *altmove_r); #endif
--- a/src/plugins/expire/expire-plugin.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/expire/expire-plugin.c Mon Jun 09 05:11:18 2008 +0300 @@ -29,6 +29,7 @@ struct expire_mailbox { union mailbox_module_context module_ctx; time_t expire_secs; + unsigned int altmove:1; }; struct expire_transaction_context { @@ -78,16 +79,16 @@ for (seq = 2; seq <= hdr->messages_count; seq++) { if (!mail_index_is_expunged(view, seq)) { mail_set_seq(mail, seq); - if (mail_get_save_date(mail, stamp_r) == 0) { - mail_free(&mail); - return; - } + if (mail_get_save_date(mail, stamp_r) == 0) + break; } } mail_free(&mail); - /* everything expunged */ - *stamp_r = 0; + if (seq > hdr->messages_count) { + /* everything expunged */ + *stamp_r = 0; + } } static int @@ -103,7 +104,9 @@ bool update_dict = FALSE; int ret; - if (xt->first_expunged) { + if (xpr_box->altmove) { + /* only moving mails - don't update the move stamps */ + } else if (xt->first_expunged) { /* first mail expunged. dict needs updating. */ first_nonexpunged_timestamp(t, &new_stamp); update_dict = TRUE; @@ -126,7 +129,8 @@ this is the first mail in the database */ ret = dict_lookup(expire.db, pool_datastack_create(), key, &value); - update_dict = ret == 0 || strtoul(value, NULL, 10) == 0; + update_dict = ret == 0 || + (ret > 0 && strtoul(value, NULL, 10) == 0); /* may not be exactly the first message's save time but a few second difference doesn't matter */ new_stamp = ioloop_time; @@ -219,7 +223,8 @@ copy(t, mail, flags, keywords, dest_mail); } -static void mailbox_expire_hook(struct mailbox *box, time_t expire_secs) +static void +mailbox_expire_hook(struct mailbox *box, time_t expire_secs, bool altmove) { struct expire_mailbox *xpr_box; @@ -233,6 +238,7 @@ box->v.save_finish = expire_save_finish; box->v.copy = expire_copy; + xpr_box->altmove = altmove; xpr_box->expire_secs = expire_secs; MODULE_CONTEXT_SET(box, expire_storage_module, xpr_box); @@ -247,15 +253,17 @@ struct mailbox *box; string_t *vname; unsigned int secs; + bool altmove; box = xpr_storage->super.mailbox_open(storage, name, input, flags); if (box != NULL) { vname = t_str_new(128); (void)mail_namespace_get_vname(storage->ns, vname, name); - secs = expire_box_find_min_secs(expire.env, str_c(vname)); + secs = expire_box_find_min_secs(expire.env, str_c(vname), + &altmove); if (secs != 0) - mailbox_expire_hook(box, secs); + mailbox_expire_hook(box, secs, altmove); } return box; } @@ -264,15 +272,15 @@ { union mail_storage_module_context *xpr_storage; - if (expire.next_hook_mail_storage_created != NULL) - expire.next_hook_mail_storage_created(storage); - xpr_storage = p_new(storage->pool, union mail_storage_module_context, 1); xpr_storage->super = storage->v; storage->v.mailbox_open = expire_mailbox_open; MODULE_CONTEXT_SET_SELF(storage, expire_storage_module, xpr_storage); + + if (expire.next_hook_mail_storage_created != NULL) + expire.next_hook_mail_storage_created(storage); } void expire_plugin_init(void) @@ -286,9 +294,11 @@ if (dict_uri == NULL) i_fatal("expire plugin: expire_dict setting missing"); + expire.username = getenv("USER"); expire.env = expire_env_init(expunge_env, altmove_env); - expire.db = dict_init(dict_uri, DICT_DATA_TYPE_UINT32, NULL); - expire.username = getenv("USER"); + expire.db = dict_init(dict_uri, DICT_DATA_TYPE_UINT32, expire.username); + if (expire.db == NULL) + i_fatal("expire plugin: dict_init() failed"); expire.next_hook_mail_storage_created = hook_mail_storage_created;
--- a/src/plugins/expire/expire-tool.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/expire/expire-tool.c Mon Jun 09 05:11:18 2008 +0300 @@ -204,6 +204,9 @@ ctx.namespace_pool = pool_alloconly_create("namespaces", 1024); env = expire_env_init(getenv("EXPIRE"), getenv("EXPIRE_ALTMOVE")); dict = dict_init(getenv("EXPIRE_DICT"), DICT_DATA_TYPE_UINT32, ""); + if (dict == NULL) + i_fatal("dict_init() failed"); + trans = dict_transaction_begin(dict); iter = dict_iterate_init(dict, DICT_PATH_SHARED, DICT_ITERATE_FLAG_SORT_BY_VALUE);
--- a/src/plugins/fts-squat/fts-backend-squat.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/fts-squat/fts-backend-squat.c Mon Jun 09 05:11:18 2008 +0300 @@ -7,6 +7,8 @@ #include "squat-trie.h" #include "fts-squat-plugin.h" +#include <stdlib.h> + #define SQUAT_FILE_PREFIX "dovecot.index.search" struct squat_fts_backend { @@ -19,12 +21,39 @@ struct squat_trie_build_context *build_ctx; }; +static void +fts_backend_squat_set(struct squat_fts_backend *backend, const char *str) +{ + const char *const *tmp; + int len; + + for (tmp = t_strsplit_spaces(str, " "); *tmp != NULL; tmp++) { + if (strncmp(*tmp, "partial=", 8) == 0) { + len = atoi(*tmp + 8); + if (len <= 0) { + i_fatal("fts_squat: Invalid partial len: %s", + *tmp + 8); + } + squat_trie_set_partial_len(backend->trie, len); + } else if (strncmp(*tmp, "full=", 5) == 0) { + len = atoi(*tmp + 5); + if (len <= 0) { + i_fatal("fts_squat: Invalid full len: %s", + *tmp + 5); + } + squat_trie_set_full_len(backend->trie, len); + } else { + i_fatal("fts_squat: Invalid setting: %s", *tmp); + } + } +} + static struct fts_backend *fts_backend_squat_init(struct mailbox *box) { struct squat_fts_backend *backend; struct mail_storage *storage; struct mailbox_status status; - const char *path; + const char *path, *env; enum squat_index_flags flags = 0; storage = mailbox_get_storage(box); @@ -50,6 +79,10 @@ squat_trie_init(t_strconcat(path, "/"SQUAT_FILE_PREFIX, NULL), status.uidvalidity, storage->lock_method, flags); + + env = getenv("FTS_SQUAT"); + if (env != NULL) + fts_backend_squat_set(backend, env); return &backend->backend; }
--- a/src/plugins/fts-squat/squat-trie-private.h Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/fts-squat/squat-trie-private.h Mon Jun 09 05:11:18 2008 +0300 @@ -133,6 +133,8 @@ size_t mmap_size; unsigned char default_normalize_map[256]; + unsigned int default_partial_len; + unsigned int default_full_len; unsigned int corrupted:1; };
--- a/src/plugins/fts-squat/squat-trie.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/fts-squat/squat-trie.c Mon Jun 09 05:11:18 2008 +0300 @@ -7,6 +7,7 @@ #include "istream.h" #include "ostream.h" #include "unichar.h" +#include "nfs-workarounds.h" #include "file-cache.h" #include "seq-range-array.h" #include "squat-uidlist.h" @@ -145,6 +146,8 @@ trie->dotlock_set.nfs_flush = (flags & SQUAT_INDEX_FLAG_NFS_FLUSH) != 0; trie->dotlock_set.timeout = SQUAT_TRIE_LOCK_TIMEOUT; trie->dotlock_set.stale_timeout = SQUAT_TRIE_DOTLOCK_STALE_TIMEOUT; + trie->default_partial_len = DEFAULT_PARTIAL_LEN; + trie->default_full_len = DEFAULT_FULL_LEN; return trie; } @@ -190,14 +193,24 @@ i_free(trie); } +void squat_trie_set_partial_len(struct squat_trie *trie, unsigned int len) +{ + trie->default_partial_len = len; +} + +void squat_trie_set_full_len(struct squat_trie *trie, unsigned int len) +{ + trie->default_full_len = len; +} + static void squat_trie_header_init(struct squat_trie *trie) { memset(&trie->hdr, 0, sizeof(trie->hdr)); trie->hdr.version = SQUAT_TRIE_VERSION; trie->hdr.indexid = time(NULL); trie->hdr.uidvalidity = trie->uidvalidity; - trie->hdr.partial_len = DEFAULT_PARTIAL_LEN; - trie->hdr.full_len = DEFAULT_FULL_LEN; + trie->hdr.partial_len = trie->default_partial_len; + trie->hdr.full_len = trie->default_full_len; i_assert(sizeof(trie->hdr.normalize_map) == sizeof(trie->default_normalize_map)); @@ -234,7 +247,9 @@ { struct stat st, st2; - if (stat(trie->path, &st) < 0) { + if ((trie->flags & SQUAT_INDEX_FLAG_NFS_FLUSH) != 0) + nfs_flush_file_handle_cache(trie->path); + if (nfs_safe_stat(trie->path, &st) < 0) { if (errno == ENOENT) return 1; @@ -242,6 +257,8 @@ return -1; } if (fstat(trie->fd, &st2) < 0) { + if (errno == ESTALE) + return 1; i_error("fstat(%s) failed: %m", trie->path); return -1; } @@ -269,7 +286,7 @@ *file_lock_r = NULL; *dotlock_r = NULL; - while (trie->fd != -1) { + for (;;) { if (trie->lock_method != FILE_LOCK_METHOD_DOTLOCK) { ret = file_wait_lock(trie->fd, trie->path, lock_type, trie->lock_method, @@ -290,7 +307,7 @@ file and try to lock again */ ret = squat_trie_is_file_stale(trie); if (ret == 0) - return 1; + break; if (*file_lock_r != NULL) file_unlock(file_lock_r); @@ -302,8 +319,13 @@ squat_trie_close(trie); if (squat_trie_open_fd(trie) < 0) return -1; + if (trie->fd == -1) + return 0; } - return 0; + + if ((trie->flags & SQUAT_INDEX_FLAG_NFS_FLUSH) != 0) + nfs_flush_read_cache_locked(trie->path, trie->fd); + return 1; } static void @@ -621,17 +643,17 @@ { struct squat_node *child; unsigned char *str; - unsigned int uid, idx, str_len = node->leaf_string_length; + unsigned int uid, idx, leafstr_len = node->leaf_string_length; - i_assert(str_len > 0); + i_assert(leafstr_len > 0); /* make a copy of the leaf string and convert to normal node by removing it. */ - str = t_malloc(str_len); + str = t_malloc(leafstr_len); if (!NODE_IS_DYNAMIC_LEAF(node)) - memcpy(str, node->children.static_leaf_string, str_len); + memcpy(str, node->children.static_leaf_string, leafstr_len); else { - memcpy(str, node->children.leaf_string, str_len); + memcpy(str, node->children.leaf_string, leafstr_len); i_free(node->children.leaf_string); } node->leaf_string_length = 0; @@ -649,16 +671,17 @@ } i_assert(!child->have_sequential && child->children.data == NULL); - if (str_len > 1) { + if (leafstr_len > 1) { /* make the child a leaf string */ - str_len--; - child->leaf_string_length = str_len; + leafstr_len--; + child->leaf_string_length = leafstr_len; if (!NODE_IS_DYNAMIC_LEAF(child)) { memcpy(child->children.static_leaf_string, - str + 1, str_len); + str + 1, leafstr_len); } else { - child->children.leaf_string = i_malloc(str_len); - memcpy(child->children.leaf_string, str + 1, str_len); + child->children.leaf_string = i_malloc(leafstr_len); + memcpy(child->children.leaf_string, + str + 1, leafstr_len); } } } @@ -669,10 +692,10 @@ const unsigned char *data, unsigned int data_len) { const unsigned char *str = NODE_LEAF_STRING(node); - const unsigned int str_len = node->leaf_string_length; + const unsigned int leafstr_len = node->leaf_string_length; unsigned int i; - if (data_len != str_len) { + if (data_len != leafstr_len) { /* different lengths, can't match */ T_BEGIN { node_split_string(ctx, node); @@ -1735,15 +1758,15 @@ return -1; } if (node->leaf_string_length != 0) { - unsigned int str_len = node->leaf_string_length; + unsigned int len = node->leaf_string_length; const unsigned char *str; - if (str_len > sizeof(node->children.static_leaf_string)) + if (len > sizeof(node->children.static_leaf_string)) str = node->children.leaf_string; else str = node->children.static_leaf_string; - if (size > str_len || memcmp(data, str, size) != 0) + if (size > len || memcmp(data, str, size) != 0) return 0; /* match */ @@ -2003,6 +2026,7 @@ squat_trie_filter_type(type, &ctx.tmp_uids, definite_uids); } + seq_range_array_remove_seq_range(maybe_uids, definite_uids); squat_trie_add_unknown(trie, maybe_uids); array_free(&ctx.tmp_uids); array_free(&ctx.tmp_uids2);
--- a/src/plugins/fts-squat/squat-trie.h Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/fts-squat/squat-trie.h Mon Jun 09 05:11:18 2008 +0300 @@ -23,6 +23,9 @@ enum squat_index_flags flags); void squat_trie_deinit(struct squat_trie **trie); +void squat_trie_set_partial_len(struct squat_trie *trie, unsigned int len); +void squat_trie_set_full_len(struct squat_trie *trie, unsigned int len); + void squat_trie_refresh(struct squat_trie *trie); int squat_trie_build_init(struct squat_trie *trie, uint32_t *last_uid_r,
--- a/src/plugins/fts/fts-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/fts/fts-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -657,17 +657,9 @@ return ret; } -void fts_mailbox_opened(struct mailbox *box) +static void fts_mailbox_init(struct mailbox *box, const char *env) { struct fts_mailbox *fbox; - const char *env; - - if (fts_next_hook_mailbox_opened != NULL) - fts_next_hook_mailbox_opened(box); - - env = getenv("FTS"); - if (env == NULL) - return; fbox = i_new(struct fts_mailbox, 1); fbox->env = env; @@ -684,3 +676,15 @@ MODULE_CONTEXT_SET(box, fts_storage_module, fbox); } + +void fts_mailbox_opened(struct mailbox *box) +{ + const char *env; + + env = getenv("FTS"); + if (env != NULL) + fts_mailbox_init(box, env); + + if (fts_next_hook_mailbox_opened != NULL) + fts_next_hook_mailbox_opened(box); +}
--- a/src/plugins/lazy-expunge/lazy-expunge-plugin.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/lazy-expunge/lazy-expunge-plugin.c Mon Jun 09 05:11:18 2008 +0300 @@ -472,7 +472,7 @@ return 0; } -static void lazy_expunge_mail_storage_created(struct mail_storage *storage) +static void lazy_expunge_mail_storage_init(struct mail_storage *storage) { struct lazy_expunge_mailbox_list *llist = LAZY_EXPUNGE_LIST_CONTEXT(storage->list); @@ -480,13 +480,6 @@ const char *const *p; unsigned int i; - if (lazy_expunge_next_hook_mail_storage_created != NULL) - lazy_expunge_next_hook_mail_storage_created(storage); - - /* only maildir supported for now */ - if (strcmp(storage->name, "maildir") != 0) - return; - /* if this is one of our internal storages, mark it as such before quota plugin sees it */ p = t_strsplit_spaces(getenv("LAZY_EXPUNGE"), " "); @@ -506,6 +499,16 @@ MODULE_CONTEXT_SET(storage, lazy_expunge_mail_storage_module, lstorage); } +static void lazy_expunge_mail_storage_created(struct mail_storage *storage) +{ + /* only maildir supported for now */ + if (strcmp(storage->name, "maildir") == 0) + lazy_expunge_mail_storage_init(storage); + + if (lazy_expunge_next_hook_mail_storage_created != NULL) + lazy_expunge_next_hook_mail_storage_created(storage); +} + static void lazy_expunge_mailbox_list_created(struct mailbox_list *list) { struct lazy_expunge_mailbox_list *llist;
--- a/src/plugins/mail-log/mail-log-plugin.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/mail-log/mail-log-plugin.c Mon Jun 09 05:11:18 2008 +0300 @@ -499,29 +499,29 @@ { union mail_storage_module_context *lstorage; - if (mail_log_next_hook_mail_storage_created != NULL) - mail_log_next_hook_mail_storage_created(storage); - lstorage = p_new(storage->pool, union mail_storage_module_context, 1); lstorage->super = storage->v; storage->v.mailbox_open = mail_log_mailbox_open; MODULE_CONTEXT_SET_SELF(storage, mail_log_storage_module, lstorage); + + if (mail_log_next_hook_mail_storage_created != NULL) + mail_log_next_hook_mail_storage_created(storage); } static void mail_log_mailbox_list_created(struct mailbox_list *list) { union mailbox_list_module_context *llist; - if (mail_log_next_hook_mailbox_list_created != NULL) - mail_log_next_hook_mailbox_list_created(list); - llist = p_new(list->pool, union mailbox_list_module_context, 1); llist->super = list->v; list->v.delete_mailbox = mail_log_mailbox_list_delete; list->v.rename_mailbox = mail_log_mailbox_list_rename; MODULE_CONTEXT_SET_SELF(list, mail_log_mailbox_list_module, llist); + + if (mail_log_next_hook_mailbox_list_created != NULL) + mail_log_next_hook_mailbox_list_created(list); } static enum mail_log_field mail_log_parse_fields(const char *str)
--- a/src/plugins/mbox-snarf/mbox-snarf-plugin.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/mbox-snarf/mbox-snarf-plugin.c Mon Jun 09 05:11:18 2008 +0300 @@ -165,9 +165,6 @@ { struct mbox_snarf_mail_storage *mstorage; - if (mbox_snarf_next_hook_mail_storage_created != NULL) - mbox_snarf_next_hook_mail_storage_created(storage); - mstorage = p_new(storage->pool, struct mbox_snarf_mail_storage, 1); mstorage->snarf_inbox_path = p_strdup(storage->pool, home_expand(getenv("MBOX_SNARF"))); @@ -175,6 +172,9 @@ storage->v.mailbox_open = mbox_snarf_mailbox_open; MODULE_CONTEXT_SET(storage, mbox_snarf_storage_module, mstorage); + + if (mbox_snarf_next_hook_mail_storage_created != NULL) + mbox_snarf_next_hook_mail_storage_created(storage); } void mbox_snarf_plugin_init(void)
--- a/src/plugins/quota/quota-count.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/quota/quota-count.c Mon Jun 09 05:11:18 2008 +0300 @@ -6,9 +6,11 @@ #include "mail-storage.h" #include "quota-private.h" -static int quota_count_mailbox(struct mail_storage *storage, const char *name, - uint64_t *bytes_r, uint64_t *count_r) +static int +quota_count_mailbox(struct quota_root *root, struct mail_storage *storage, + const char *name, uint64_t *bytes_r, uint64_t *count_r) { + struct quota_rule *rule; struct mailbox *box; struct mailbox_transaction_context *trans; struct mail_search_context *ctx; @@ -17,6 +19,12 @@ uoff_t size; int ret = 0; + rule = quota_root_rule_find(root, name); + if (rule != NULL && rule->ignore) { + /* mailbox not included in quota */ + return 0; + } + box = mailbox_open(storage, name, NULL, MAILBOX_OPEN_READONLY | MAILBOX_OPEN_KEEP_RECENT); if (box == NULL) @@ -53,8 +61,9 @@ return ret; } -static int quota_count_storage(struct mail_storage *storage, - uint64_t *bytes, uint64_t *count) +static int +quota_count_storage(struct quota_root *root, struct mail_storage *storage, + uint64_t *bytes, uint64_t *count) { struct mailbox_list_iterate_context *ctx; const struct mailbox_info *info; @@ -65,7 +74,7 @@ while ((info = mailbox_list_iter_next(ctx)) != NULL) { if ((info->flags & (MAILBOX_NONEXISTENT | MAILBOX_NOSELECT)) == 0) { - ret = quota_count_mailbox(storage, info->name, + ret = quota_count_mailbox(root, storage, info->name, bytes, count); if (ret < 0) break; @@ -77,7 +86,7 @@ return ret; } -int quota_count(struct quota *quota, uint64_t *bytes_r, uint64_t *count_r) +int quota_count(struct quota_root *root, uint64_t *bytes_r, uint64_t *count_r) { struct mail_storage *const *storages; unsigned int i, count; @@ -85,9 +94,9 @@ *bytes_r = *count_r = 0; - storages = array_get("a->storages, &count); + storages = array_get(&root->quota->storages, &count); for (i = 0; i < count; i++) { - ret = quota_count_storage(storages[i], bytes_r, count_r); + ret = quota_count_storage(root, storages[i], bytes_r, count_r); if (ret < 0) break; }
--- a/src/plugins/quota/quota-dict.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/quota/quota-dict.c Mon Jun 09 05:11:18 2008 +0300 @@ -78,7 +78,7 @@ struct dict_transaction_context *dt; uint64_t bytes, count; - if (quota_count(root->root.quota, &bytes, &count) < 0) + if (quota_count(&root->root, &bytes, &count) < 0) return -1; T_BEGIN {
--- a/src/plugins/quota/quota-fs.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/quota/quota-fs.c Mon Jun 09 05:11:18 2008 +0300 @@ -203,7 +203,7 @@ /* if there are more unused quota roots, copy this mount to them */ roots = array_get(&root->root.quota->roots, &count); for (i = 0; i < count; i++) { - struct fs_quota_root *root = (struct fs_quota_root *)roots[i]; + root = (struct fs_quota_root *)roots[i]; if (QUOTA_ROOT_MATCH(root, mount) && root->mount == NULL) { mount->refcount++; root->mount = mount;
--- a/src/plugins/quota/quota-maildir.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/quota/quota-maildir.c Mon Jun 09 05:11:18 2008 +0300 @@ -36,6 +36,7 @@ struct maildir_list_context { struct mail_storage *storage; + struct maildir_quota_root *root; struct mailbox_list_iterate_context *iter; const struct mailbox_info *info; @@ -121,12 +122,14 @@ } static struct maildir_list_context * -maildir_list_init(struct mail_storage *storage) +maildir_list_init(struct maildir_quota_root *root, + struct mail_storage *storage) { struct maildir_list_context *ctx; ctx = i_new(struct maildir_list_context, 1); ctx->storage = storage; + ctx->root = root; ctx->path = str_new(default_pool, 512); ctx->iter = mailbox_list_iter_init(mail_storage_get_list(storage), "*", MAILBOX_LIST_ITER_RETURN_NO_FLAGS); @@ -136,6 +139,7 @@ static const char * maildir_list_next(struct maildir_list_context *ctx, time_t *mtime_r) { + struct quota_rule *rule; struct stat st; bool is_file; @@ -144,6 +148,13 @@ ctx->info = mailbox_list_iter_next(ctx->iter); if (ctx->info == NULL) return NULL; + + rule = quota_root_rule_find(&ctx->root->root, + ctx->info->name); + if (rule != NULL && rule->ignore) { + /* mailbox not included in quota */ + continue; + } } T_BEGIN { @@ -185,13 +196,14 @@ } static int -maildirs_check_have_changed(struct mail_storage *storage, time_t latest_mtime) +maildirs_check_have_changed(struct maildir_quota_root *root, + struct mail_storage *storage, time_t latest_mtime) { struct maildir_list_context *ctx; time_t mtime; int ret = 0; - ctx = maildir_list_init(storage); + ctx = maildir_list_init(root, storage); while (maildir_list_next(ctx, &mtime) != NULL) { if (mtime > latest_mtime) { ret = 1; @@ -270,7 +282,7 @@ time_t mtime; int ret = 0; - ctx = maildir_list_init(storage); + ctx = maildir_list_init(root, storage); while ((dir = maildir_list_next(ctx, &mtime)) != NULL) { if (mtime > root->recalc_last_stamp) root->recalc_last_stamp = mtime; @@ -332,7 +344,7 @@ if (ret == 0) { /* check if any of the directories have changed */ for (i = 0; i < count; i++) { - ret = maildirs_check_have_changed(storages[i], + ret = maildirs_check_have_changed(root, storages[i], root->recalc_last_stamp); if (ret != 0) break; @@ -726,8 +738,13 @@ /* no limits */ return 0; } - /* make sure the latest file is opened. */ - (void)maildirsize_open(root); + + /* even though we don't really care about the limits in here ourself, + we do want to make sure the header gets updated if the limits have + changed. also this makes sure the maildirsize file is created if + it doesn't exist. */ + if (maildirquota_refresh(root) < 0) + return -1; if (root->fd == -1 || ctx->recalculate || maildirsize_update(root, ctx->count_used, ctx->bytes_used) < 0)
--- a/src/plugins/quota/quota-plugin.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/quota/quota-plugin.c Mon Jun 09 05:11:18 2008 +0300 @@ -31,8 +31,8 @@ break; if (quota_root_add_rule(root, rule, &error) < 0) { - i_fatal("Quota root %s: Invalid rule: %s", - root_name, rule); + i_fatal("Quota root %s: Invalid rule %s: %s", + root_name, rule, error); } rule_name = t_strdup_printf("%s_RULE%d", root_name, i); }
--- a/src/plugins/quota/quota-private.h Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/quota/quota-private.h Mon Jun 09 05:11:18 2008 +0300 @@ -25,6 +25,9 @@ int64_t bytes_limit, count_limit; /* relative to default_rule */ unsigned int bytes_percent, count_percent; + + /* Don't include this mailbox in quota */ + unsigned int ignore:1; }; struct quota_warning_rule { @@ -103,7 +106,10 @@ void quota_remove_user_storage(struct quota *quota, struct mail_storage *storage); +struct quota_rule * +quota_root_rule_find(struct quota_root *root, const char *name); + void quota_root_recalculate_relative_rules(struct quota_root *root); -int quota_count(struct quota *quota, uint64_t *bytes_r, uint64_t *count_r); +int quota_count(struct quota_root *root, uint64_t *bytes_r, uint64_t *count_r); #endif
--- a/src/plugins/quota/quota-storage.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/quota/quota-storage.c Mon Jun 09 05:11:18 2008 +0300 @@ -455,9 +455,6 @@ struct quota_mailbox_list *qlist = QUOTA_LIST_CONTEXT(storage->list); union mail_storage_module_context *qstorage; - if (quota_next_hook_mail_storage_created != NULL) - quota_next_hook_mail_storage_created(storage); - qlist->storage = storage; qstorage = p_new(storage->pool, union mail_storage_module_context, 1); @@ -472,18 +469,21 @@ /* register to user's quota roots */ quota_add_user_storage(quota_set, storage); } + + if (quota_next_hook_mail_storage_created != NULL) + quota_next_hook_mail_storage_created(storage); } void quota_mailbox_list_created(struct mailbox_list *list) { struct quota_mailbox_list *qlist; - if (quota_next_hook_mailbox_list_created != NULL) - quota_next_hook_mailbox_list_created(list); - qlist = p_new(list->pool, struct quota_mailbox_list, 1); qlist->module_ctx.super = list->v; list->v.delete_mailbox = quota_mailbox_list_delete; MODULE_CONTEXT_SET(list, quota_mailbox_list_module, qlist); + + if (quota_next_hook_mailbox_list_created != NULL) + quota_next_hook_mailbox_list_created(list); }
--- a/src/plugins/quota/quota.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/quota/quota.c Mon Jun 09 05:11:18 2008 +0300 @@ -176,7 +176,7 @@ pool_unref(&pool); } -static struct quota_rule * +struct quota_rule * quota_root_rule_find(struct quota_root *root, const char *name) { struct quota_rule *rules; @@ -353,8 +353,17 @@ } } + if (strcmp(p, "ignore") == 0) { + rule->ignore = TRUE; + if (root->quota->debug) { + i_info("Quota rule: root=%s mailbox=%s ignored", + root->name, mailbox_name); + } + return 0; + } + if (strncmp(p, "backend=", 8) == 0) { - if (!root->backend.v.parse_rule(root, rule, p, error_r)) + if (!root->backend.v.parse_rule(root, rule, p + 8, error_r)) ret = -1; } else { bool allow_negative = rule != &root->default_rule; @@ -368,7 +377,7 @@ if (root->quota->debug) { i_info("Quota rule: root=%s mailbox=%s " "bytes=%lld (%u%%) messages=%lld (%u%%)", root->name, - rule->mailbox_name != NULL ? rule->mailbox_name : "", + mailbox_name, (long long)rule->bytes_limit, rule->bytes_percent, (long long)rule->count_limit, rule->count_percent); } @@ -393,8 +402,13 @@ rule = quota_root_rule_find(root, mailbox_name); if (rule != NULL) { - bytes_limit += rule->bytes_limit; - count_limit += rule->count_limit; + if (!rule->ignore) { + bytes_limit += rule->bytes_limit; + count_limit += rule->count_limit; + } else { + bytes_limit = 0; + count_limit = 0; + } found = TRUE; } @@ -724,8 +738,10 @@ int quota_transaction_commit(struct quota_transaction_context **_ctx) { struct quota_transaction_context *ctx = *_ctx; + struct quota_rule *rule; struct quota_root *const *roots; unsigned int i, count; + const char *mailbox_name; int ret = 0; *_ctx = NULL; @@ -734,8 +750,15 @@ ret = -1; else if (ctx->bytes_used != 0 || ctx->count_used != 0 || ctx->recalculate) { + mailbox_name = mailbox_get_name(ctx->box); roots = array_get(&ctx->quota->roots, &count); for (i = 0; i < count; i++) { + rule = quota_root_rule_find(roots[i], mailbox_name); + if (rule != NULL && rule->ignore) { + /* mailbox not included in quota */ + continue; + } + if (roots[i]->backend.v.update(roots[i], ctx) < 0) ret = -1; }
--- a/src/plugins/zlib/zlib-plugin.c Sat May 17 17:50:54 2008 +0300 +++ b/src/plugins/zlib/zlib-plugin.c Mon Jun 09 05:11:18 2008 +0300 @@ -26,18 +26,32 @@ &mail_storage_module_register); static MODULE_CONTEXT_DEFINE_INIT(zlib_mail_module, &mail_module_register); +static int zlib_mail_is_compressed(struct istream *mail) +{ + const unsigned char *zheader; + size_t header_size; + + /* Peek in to the mail and see if it looks like it's compressed + (it has the correct 2 byte zlib header). This also means that users + can try to exploit security holes in zlib by APPENDing a specially + crafted mail. So let's hope zlib is free of holes. */ + if (i_stream_read_data(mail, &zheader, &header_size, 2) <= 0) + return 0; + i_stream_seek(mail, 0); + + return header_size >= 2 && zheader && + zheader[0] == 31 && zheader[1] == 139; +} + static int zlib_maildir_get_stream(struct mail *_mail, struct message_size *hdr_size, struct message_size *body_size, struct istream **stream_r) { - struct maildir_mailbox *mbox = (struct maildir_mailbox *)_mail->box; struct mail_private *mail = (struct mail_private *)_mail; struct index_mail *imail = (struct index_mail *)mail; union mail_module_context *zmail = ZLIB_MAIL_CONTEXT(mail); struct istream *input; - const char *fname, *p; - enum maildir_uidlist_rec_flag flags; int fd; if (imail->data.stream != NULL) { @@ -49,10 +63,7 @@ return -1; i_assert(input == imail->data.stream); - fname = maildir_uidlist_lookup(mbox->uidlist, _mail->uid, &flags); - p = fname == NULL ? NULL : strstr(fname, ":2,"); - if (p != NULL && strchr(p + 3, 'Z') != NULL) { - /* has a Z flag - it's compressed */ + if (zlib_mail_is_compressed(imail->data.stream)) { fd = dup(i_stream_get_fd(imail->data.stream)); if (fd == -1) i_error("zlib plugin: dup() failed: %m"); @@ -140,14 +151,14 @@ { union mail_storage_module_context *qstorage; - if (zlib_next_hook_mail_storage_created != NULL) - zlib_next_hook_mail_storage_created(storage); - qstorage = p_new(storage->pool, union mail_storage_module_context, 1); qstorage->super = storage->v; storage->v.mailbox_open = zlib_mailbox_open; MODULE_CONTEXT_SET_SELF(storage, zlib_storage_module, qstorage); + + if (zlib_next_hook_mail_storage_created != NULL) + zlib_next_hook_mail_storage_created(storage); } void zlib_plugin_init(void)
--- a/src/pop3-login/client-authenticate.c Sat May 17 17:50:54 2008 +0300 +++ b/src/pop3-login/client-authenticate.c Mon Jun 09 05:11:18 2008 +0300 @@ -170,7 +170,7 @@ } client_send_line(client, "+OK Logged in."); - client_destroy(client, "Login"); + client_destroy_success(client, "Login"); break; case SASL_SERVER_REPLY_AUTH_FAILED: case SASL_SERVER_REPLY_CLIENT_ERROR: @@ -197,7 +197,7 @@ else { client_send_line(client, t_strconcat("-ERR [IN-USE] ", data, NULL)); - client_destroy(client, data); + client_destroy_success(client, data); } break; case SASL_SERVER_REPLY_CONTINUE:
--- a/src/pop3-login/client.c Sat May 17 17:50:54 2008 +0300 +++ b/src/pop3-login/client.c Mon Jun 09 05:11:18 2008 +0300 @@ -150,9 +150,7 @@ client_destroy(client, "Aborted login " "(tried to use disabled plaintext authentication)"); } else { - client_destroy(client, t_strdup_printf( - "Aborted login (%u authentication attempts)", - client->common.auth_attempts)); + client_destroy(client, "Aborted login"); } return TRUE; } @@ -341,12 +339,22 @@ return &client->common; } +void client_destroy_success(struct pop3_client *client, const char *reason) +{ + client->login_success = TRUE; + client_destroy(client, reason); +} + void client_destroy(struct pop3_client *client, const char *reason) { if (client->destroyed) return; client->destroyed = TRUE; + if (!client->login_success && reason != NULL) { + reason = t_strdup_printf("%s (auth failed, %u attempts)", + reason, client->common.auth_attempts); + } if (reason != NULL) client_syslog(&client->common, reason);
--- a/src/pop3-login/client.h Sat May 17 17:50:54 2008 +0300 +++ b/src/pop3-login/client.h Mon Jun 09 05:11:18 2008 +0300 @@ -28,12 +28,14 @@ char *apop_challenge; struct auth_connect_id auth_id; + unsigned int login_success:1; unsigned int authenticating:1; unsigned int auth_connected:1; unsigned int destroyed:1; }; void client_destroy(struct pop3_client *client, const char *reason); +void client_destroy_success(struct pop3_client *client, const char *reason); void client_destroy_internal_failure(struct pop3_client *client); void client_send_line(struct pop3_client *client, const char *line);
--- a/src/pop3-login/pop3-proxy.c Sat May 17 17:50:54 2008 +0300 +++ b/src/pop3-login/pop3-proxy.c Mon Jun 09 05:11:18 2008 +0300 @@ -32,7 +32,7 @@ /* failed for some reason, probably server disconnected */ client_send_line(client, "-ERR [IN-USE] Temporary login failure."); - client_destroy(client, NULL); + client_destroy_success(client, NULL); return; } @@ -47,7 +47,7 @@ return; case -1: /* disconnected */ - client_destroy(client, "Proxy: Remote disconnected"); + client_destroy_success(client, "Proxy: Remote disconnected"); return; } @@ -99,8 +99,8 @@ break; /* Login successful. Send this line to client. */ + line = t_strconcat(line, "\r\n", NULL); (void)o_stream_send_str(client->output, line); - (void)o_stream_send(client->output, "\r\n", 2); msg = t_strdup_printf("proxy(%s): started proxying to %s:%u", client->common.virtual_user, @@ -114,7 +114,7 @@ client->input = NULL; client->output = NULL; client->common.fd = -1; - client_destroy(client, msg); + client_destroy_success(client, msg); return; }
--- a/src/pop3/client.c Sat May 17 17:50:54 2008 +0300 +++ b/src/pop3/client.c Mon Jun 09 05:11:18 2008 +0300 @@ -241,11 +241,20 @@ return str_c(str); } +static const char *client_get_disconnect_reason(struct client *client) +{ + errno = client->input->stream_errno != 0 ? + client->input->stream_errno : + client->output->stream_errno; + return errno == 0 || errno == EPIPE ? "Connection closed" : + t_strdup_printf("Connection closed: %m"); +} + void client_destroy(struct client *client, const char *reason) { if (!client->disconnected) { if (reason == NULL) - reason = "Disconnected"; + reason = client_get_disconnect_reason(client); i_info("%s %s", reason, client_stats(client)); }
--- a/src/pop3/commands.c Sat May 17 17:50:54 2008 +0300 +++ b/src/pop3/commands.c Mon Jun 09 05:11:18 2008 +0300 @@ -507,6 +507,49 @@ struct mail_search_arg search_arg; }; +static void pop3_get_uid(struct cmd_uidl_context *ctx, + struct var_expand_table *tab, string_t *str) +{ + char uid_str[MAX_INT_STRLEN]; + const char *uidl; + + if (mail_get_special(ctx->mail, MAIL_FETCH_UIDL_BACKEND, &uidl) == 0 && + *uidl != '\0') { + str_append(str, uidl); + return; + } + + if (reuse_xuidl && + mail_get_first_header(ctx->mail, "X-UIDL", &uidl) > 0) { + str_append(str, uidl); + return; + } + + if ((uidl_keymask & UIDL_UID) != 0) { + i_snprintf(uid_str, sizeof(uid_str), "%u", + ctx->mail->uid); + tab[1].value = uid_str; + } + if ((uidl_keymask & UIDL_MD5) != 0) { + if (mail_get_special(ctx->mail, MAIL_FETCH_HEADER_MD5, + &tab[2].value) < 0 || + *tab[2].value == '\0') { + /* broken */ + i_fatal("UIDL: Header MD5 not found"); + } + } + if ((uidl_keymask & UIDL_FILE_NAME) != 0) { + if (mail_get_special(ctx->mail, + MAIL_FETCH_UIDL_FILE_NAME, + &tab[3].value) < 0 || + *tab[3].value == '\0') { + /* broken */ + i_fatal("UIDL: File name not found"); + } + } + var_expand(str, uidl_format, tab); +} + static bool list_uids_iter(struct client *client, struct cmd_uidl_context *ctx) { static struct var_expand_table static_tab[] = { @@ -518,8 +561,6 @@ }; struct var_expand_table *tab; string_t *str; - char uid_str[MAX_INT_STRLEN]; - const char *uidl; int ret; bool found = FALSE; @@ -537,38 +578,11 @@ } found = TRUE; - if ((uidl_keymask & UIDL_UID) != 0) { - i_snprintf(uid_str, sizeof(uid_str), "%u", - ctx->mail->uid); - tab[1].value = uid_str; - } - if ((uidl_keymask & UIDL_MD5) != 0) { - if (mail_get_special(ctx->mail, MAIL_FETCH_HEADER_MD5, - &tab[2].value) < 0 || - *tab[2].value == '\0') { - /* broken */ - i_fatal("UIDL: Header MD5 not found"); - } - } - if ((uidl_keymask & UIDL_FILE_NAME) != 0) { - if (mail_get_special(ctx->mail, - MAIL_FETCH_UIDL_FILE_NAME, - &tab[3].value) < 0 || - *tab[3].value == '\0') { - /* broken */ - i_fatal("UIDL: File name not found"); - } - } - str_truncate(str, 0); str_printfa(str, ctx->message == 0 ? "%u " : "+OK %u ", ctx->mail->seq); + pop3_get_uid(ctx, tab, str); - if (reuse_xuidl && - mail_get_first_header(ctx->mail, "X-UIDL", &uidl) > 0) - str_append(str, uidl); - else - var_expand(str, uidl_format, tab); ret = client_send_line(client, "%s", str_c(str)); if (ret < 0) break;
--- a/src/tests/Makefile.am Sat May 17 17:50:54 2008 +0300 +++ b/src/tests/Makefile.am Mon Jun 09 05:11:18 2008 +0300 @@ -44,4 +44,6 @@ $(RAND_LIBS) \ libtest.a \ ../lib-imap/libimap.a \ + ../lib-mail/libmail.a \ + ../lib-charset/libcharset.a \ ../lib/liblib.a
--- a/src/tests/test-lib.c Sat May 17 17:50:54 2008 +0300 +++ b/src/tests/test-lib.c Mon Jun 09 05:11:18 2008 +0300 @@ -6,6 +6,7 @@ #include "base64.h" #include "bsearch-insert-pos.h" #include "aqueue.h" +#include "network.h" #include "priorityq.h" #include "seq-range-array.h" #include "str-sanitize.h" @@ -143,6 +144,7 @@ buf = buffer_create_dynamic(default_pool, 1); for (i = 0; i < BUF_TEST_SIZE; i++) testdata[i] = random(); + memset(shadowbuf, 0, sizeof(shadowbuf)); srand(1); shadowbuf_size = 0; @@ -366,6 +368,50 @@ test_out("mempool_alloconly", success); } +struct test_net_is_in_network_input { + const char *ip; + const char *net; + unsigned int bits; + bool ret; +}; + +static void test_net_is_in_network(void) +{ + static struct test_net_is_in_network_input input[] = { + { "1.2.3.4", "1.2.3.4", 32, TRUE }, + { "1.2.3.4", "1.2.3.3", 32, FALSE }, + { "1.2.3.4", "1.2.3.5", 32, FALSE }, + { "1.2.3.4", "1.2.2.4", 32, FALSE }, + { "1.2.3.4", "1.1.3.4", 32, FALSE }, + { "1.2.3.4", "0.2.3.4", 32, FALSE }, + { "1.2.3.253", "1.2.3.254", 31, FALSE }, + { "1.2.3.254", "1.2.3.254", 31, TRUE }, + { "1.2.3.255", "1.2.3.254", 31, TRUE }, + { "1.2.3.255", "1.2.3.0", 24, TRUE }, + { "1.2.255.255", "1.2.254.0", 23, TRUE }, + { "255.255.255.255", "128.0.0.0", 1, TRUE }, + { "255.255.255.255", "127.0.0.0", 1, FALSE } +#ifdef HAVE_IPV6 + , + { "1234:5678::abcf", "1234:5678::abce", 127, TRUE }, + { "1234:5678::abcd", "1234:5678::abce", 127, FALSE }, + { "123e::ffff", "123e::0", 15, TRUE }, + { "123d::ffff", "123e::0", 15, FALSE } +#endif + }; + struct ip_addr ip, net_ip; + unsigned int i; + bool success; + + for (i = 0; i < N_ELEMENTS(input); i++) { + net_addr2ip(input[i].ip, &ip); + net_addr2ip(input[i].net, &net_ip); + success = net_is_in_network(&ip, &net_ip, input[i].bits) == + input[i].ret; + test_out(t_strdup_printf("net_is_in_network(%u)", i), success); + } +} + struct pq_test_item { struct priorityq_item item; int num; @@ -680,6 +726,7 @@ test_bsearch_insert_pos, test_buffer, test_mempool_alloconly, + test_net_is_in_network, test_priorityq, test_seq_range_array, test_str_sanitize,
--- a/src/util/idxview.c Sat May 17 17:50:54 2008 +0300 +++ b/src/util/idxview.c Mon Jun 09 05:11:18 2008 +0300 @@ -17,17 +17,18 @@ struct maildir_index_header { uint32_t new_check_time, new_mtime, new_mtime_nsecs; uint32_t cur_check_time, cur_mtime, cur_mtime_nsecs; + uint32_t uidlist_mtime, uidlist_mtime_nsecs, uidlist_size; }; struct dbox_index_header { uint32_t last_dirty_flush_stamp; }; -static const char *unixdate2str(time_t time) +static const char *unixdate2str(time_t timestamp) { static char buf[64]; struct tm *tm; - tm = localtime(&time); + tm = localtime(×tamp); strftime(buf, sizeof(buf), "%Y-%m-%d %H:%M", tm); return buf; } @@ -80,12 +81,15 @@ const struct maildir_index_header *hdr = data; printf("header\n"); - printf(" - new_check_time = %s\n", unixdate2str(hdr->new_check_time)); - printf(" - new_mtime ..... = %s\n", unixdate2str(hdr->new_mtime)); - printf(" - new_mtime_nsecs = %u\n", hdr->new_mtime_nsecs); - printf(" - cur_check_time = %s\n", unixdate2str(hdr->cur_check_time)); - printf(" - cur_mtime ..... = %s\n", unixdate2str(hdr->cur_mtime)); - printf(" - cur_mtime_nsecs = %u\n", hdr->cur_mtime_nsecs); + printf(" - new_check_time .... = %s\n", unixdate2str(hdr->new_check_time)); + printf(" - new_mtime ......... = %s\n", unixdate2str(hdr->new_mtime)); + printf(" - new_mtime_nsecs ... = %u\n", hdr->new_mtime_nsecs); + printf(" - cur_check_time .... = %s\n", unixdate2str(hdr->cur_check_time)); + printf(" - cur_mtime ......... = %s\n", unixdate2str(hdr->cur_mtime)); + printf(" - cur_mtime_nsecs.... = %u\n", hdr->cur_mtime_nsecs); + printf(" - uidlist_mtime ..... = %s\n", unixdate2str(hdr->uidlist_mtime)); + printf(" - uidlist_mtime_nsecs = %u\n", hdr->uidlist_mtime_nsecs); + printf(" - uidlist_size ...... = %u\n", hdr->uidlist_size); } else if (strcmp(ext->name, "dbox-hdr") == 0) { const struct dbox_index_header *hdr = data;
--- a/src/util/logview.c Sat May 17 17:50:54 2008 +0300 +++ b/src/util/logview.c Mon Jun 09 05:11:18 2008 +0300 @@ -171,6 +171,7 @@ const struct mail_transaction_ext_reset *reset = data; printf(" - new_reset_id = %u\n", reset->new_reset_id); + printf(" - preserve_data = %u\n", reset->preserve_data); break; } case MAIL_TRANSACTION_EXT_HDR_UPDATE: {
--- a/src/util/rawlog.c Sat May 17 17:50:54 2008 +0300 +++ b/src/util/rawlog.c Mon Jun 09 05:11:18 2008 +0300 @@ -274,15 +274,15 @@ static void rawlog_open(enum rawlog_flags flags) { - const char *chroot, *home, *path; + const char *chroot_dir, *home, *path; struct stat st; int sfd[2]; pid_t pid; - chroot = getenv("RESTRICT_CHROOT"); + chroot_dir = getenv("RESTRICT_CHROOT"); home = getenv("HOME"); - if (chroot != NULL) - home = t_strconcat(chroot, home, NULL); + if (chroot_dir != NULL) + home = t_strconcat(chroot_dir, home, NULL); else if (home == NULL) home = "."; @@ -296,9 +296,9 @@ if (!S_ISDIR(st.st_mode)) return; - if (chroot != NULL) { + if (chroot_dir != NULL) { /* we'll chroot soon. skip over the chroot in the path. */ - path += strlen(chroot); + path += strlen(chroot_dir); } if (socketpair(AF_UNIX, SOCK_STREAM, 0, sfd) < 0)