view TODO @ 9595:166a188f9165 HEAD

mbox: Fields weren't being added to cache file during message saving.
author Timo Sirainen <>
date Fri, 30 Jul 2010 16:55:58 +0100
parents fa4b9d520687
line wrap: on
line source

 - antispam plugin: deleting mails from spam mailbox causes
   "dovecot.index reset, view is now inconsistent"
 - acl: mail_debug=yes could log something useful
 - dict quota: syncing may cause quota recalculation, which in turn syncs
   all mailboxes and then we'll assert-crash to avoid infinite looping

 - acl: users are never dropped from acl_shared_dict.
 - index-sync.c: line 39 (index_mailbox_set_recent_uid): assertion failed: (seq_range_exists(&ibox->recent_flags, uid))
    ^ when stress testing and around "Duplicate file entry" errors
 - convert plugin: convert_pop3_uidl_format setting? so old %f uidls could be
   converted to dbox..
 - proxying: support fallbacking to local (or other?) server if the first
   one is down
 - fts-solr: handle DELETE, RENAME
 - fsck -> log_file_tail_offset 2273345664 -> 996 ->
   mail-transaction-log.c: line 341 (mail_transaction_log_set_mailbox_sync_pos):
   assertion failed: (file_offset >= log->head->saved_tail_offset)
 - virtual: "Searched n% of the mailbox" gives broken numbers since
   ctx->seq jumps around. And why is it also returned when fts is enabled
   along with "Indexed n% of the mailbox"?
 - how do shared mailboxes work with plugins?
    - expire: not too well. would require knowing the mapping between shared
      namespace and the original user's namespace to avoid duplication.
    - lazy-expunge, fts, etc.?
 - dovecot-acl-list:
    - how does it work with global acls?
    - update immediately after SETACL: add/remove entries, update timestamps
    - read the entire file to memory only once and keep it there, stat() later
      to see if it has changed. if not, perhaps don't even bother stat()ing
      dovecot-acl files? at least not that often..
    - when reading dovecot-acl file and seeing a +l right on a mailbox not
      listed in dovecot-acl-list, recreate it
 - add anonymous environment for anon logins
 - fs quota: getquotaroot inbox vs. other-box should return different quotas
   if two quotas are defined
 - deliver: log mailbox name using utf8, not mutf7
 - new primes code: are hash tables now being resized too often?
 - auth_log_prefix setting similar to mail_log_prefix
 - LDAP attrs: uid=foo,uid=bar doesn't work

 - easily limit master users to be able to only log as other users within
   their domain
 - thread indexes: if we expunge a duplicate message-id: and we have a sibling
   with identical message-id:, we can probably just move the children?
   (unless there are non-sibling duplicates)
 - SEARCH INTHREAD requires no thread sorting by date - don't do it
 - CONDSTORE: use per-flag/per-keyword conflict checking
 - QRESYNC: Drop expunges from the middle of given seq sets if possible
 - use universal hash functions?

 - expire plugin: log more with mail_debug=yes
  - expire-tool -v could log UID and expire timestamps and what messages got
 - fts-squat: support ORs
 - UIDVALIDITY changed while saving -> sync errors
   - mbox: copy to Trash, manually delete copied msg, change uidvalidity,
     set nextuid=1, copy again -> error
   - recent_uids assert at least with mbox
 - quota fs: Should values returned by quota be divided by the actual
   filesystem block size instead of hardcoded DEV_BSIZE? not with AIX..
 - mailbox list fs: Listing subscriptions with children return options doesn't
   work unless iter_is_mailbox() returns the children flags
 - sieve-cmu.c crash: i_assert(buf->used - 1 == part->body_size.physical_size);
 - convert plugin: Create a r/w lock for a file. It's read-locked if
   conversion isn't wanted and released when process dies. If conversion is
   wanted and write-lock succeeds, conversion is done, if write-lock doesn't
   succeed it fallbacks to using the old storage. When process is exiting it
   again tries to write-lock and do the conversion. Add a parameter that
   specifies if conversion should be done.
 - lucene: handle replacement chars?
 - squat:
   - wrong indexid
   - fts_build_init() assertion failed: (last_uid < last_uid_locked)
   - is locking done right? it reads header without file being locked?
   - split after ~8 bytes?
   - expunges are delayed until more mails are added
 - test replacement chars (SEARCH / SORT / Squat)

 - dbox:
    - "File unexpectedly lost" doesn't get fixed by itself
    - Fix support for multi-message files
    - Delete dovecot-keywords and dovecot-uidlist after all maildir files
      have been converted to native dbox
 - DEBUG: buffer overflow checking code probably doesn't handle a successful
   t_try_realloc() or pool_alloconly_realloc() properly
 - cache: compress when we can drop temporary fields.
 - new %modifier for reverse DNS lookups with a cache
   - auth_gssapi_hostname = %Xl
   - proxying would also want DNS lookups, but not reverse..
 - ldap:
   - domain lookups which set the base for user lookup
   - same attribute can't be used for multiple values.
   - multiple attributes can't be merged to a single value.
   - implement something like:
       user_attrs {
	 uid = %{ldap:uidNumber}
	 home = %{ldap:homeDirectory}
	 quota_bytes = *:bytes=%{ldap:quota}

 - Per-user options:
   - Deny deleting non-empty mailboxes
   - Disable IDLE "still here" notifications

 - maildir+pop3/deliver fast updates:
   - with locking enabled, pop3 could just keep the one and same sync lock and
     do the whole thing using sync transaction
   - don't update dovecot-uidlist if dovecot.index.cache doesn't exist /
     there's nothing to cache
   - if all messages are expunged and there are no unknown extensions in index,
     unlink dovecot.index and rotate log and add some initial useful info to
     the log (uidvalidity, nextuid)

 - maildir
   - don't allow more than 26 keywords
   - physical separator could be configurable
   - maildir_copy_with_hardlinks: We're currently first hardlinking to tmp/ and
     then rename()ing. This wouldn't be necessary if uidlist syncing noticed
     that someone else already had added them to uidlist, and the existing UIDs
     could be assigned to them in the index.
       - copying should copy already-cached data
   - maildir_copy_preserve_filename=yes has a race condition causing "Append with
     UID n, but next_uid = y" errors when quota plugin is loaded. Practically
     won't happen except in stress testing.

 - mbox
   - UID renumbering doesn't really work after all?
   - still problems with CRLF mboxes..

 - proxy: If remote server disconnects on login:
   login: tried to change state 2 -> 2
 - logging consistency:
 - EXPUNGE command in read-only mailbox should give an error message if
   there are messages marked as \Deleted?
 - dovecot -o setting=something overriding
 - file_cache: we're growing the mmap in page size blocks, which is horribly
   slow if mremap() doesn't exist.
 - login_max_processes_count shouldn't count proxying processes

 - Allow %variables in mail_chroot setting
 - ssl_verify_client_cert isn't working if the SSL cert doesn't have CRL

 - keywords:
    - add some limits to how many there can be
       - don't return \* in PERMANENTFLAGS when we're full
    - remove unused keywords?

 - caching
    - force bits should be used only for nonregistered fields
    - change envelope parsing not to use get_headers() so imap.envelope can
      actually be cached without all the headers..
    - if there's no other pressure for compression, we should do it when
      enough temp fields are ready to be dropped
    - we could try compressing same field values into a single
      location in cache file.
    - place some maximum limit of fields to cache file? maybe some soft and
      hard limits, so when soft limit is reached drop fields that have
      been used only once. when hard limit is reached drop any fields to get
      more space. all this to avoid cache file growing infinitely.

 - mbox
    - syncing existing indexes takes 4x longer than creating new one, why?
    - how well does dirty sync + status work? it reads the last mail every
      time? not very good..
    - always add empty line. make the parser require it too? syncing should
      make sure there always exists two LFs at end of file. raw-mbox-stream
      should make sure the last message ends with LF even if it doesn't exist
      in the file
    - Quote "From ", unquote ">From "
    - COPY doesn't work to itself (lock assert crash, for now just disallowed)

 - index
    - read-only support for mailboxes where we don't have write-access
    - index file format changes:
	- pack UIDs to beginning of file with UID ranges
	- use squat-like compressed uid ranges everywhere
        - write first extension intros in dovecot.index.log always with names
	   - or better yet, drop the intro concept completely as it is now
	- add "transaction boundaries" so we know which records belong to a
	  single transaction.
	   - only after that we can remove the transaction log offset
	     overwriting (otherwise we can get partial transactions in views)

 - namespaces
    - namespaces: add new "auto_disable" flag so if the mailbox can't be opened
      (eg. file doesn't exist), just ignore the problem and disable the

 - lib-storage
    - rename: allow moving between storages, as long as they're of same type
    - x search charset asdf all -> should fail

 - login
    - imap-login: Master sent reply with unknown tag 1. client closed
      connection at the exact same time master was logging it in?
      see master_request_abort()
    - Digest-MD5: support integrity protection, and maybe crypting. Do it
      through login process like SSL is done?
    -  x login foo bar
       x NO Authentication failed.
       x login cras pass
       * BYE Disconnected for inactivity.
	^ but it's not disconnecting! (buggy dovecot-auth not replying)
	  probably because userdb lookup didn't reply, and fd was already sent
	  for master.. should imap-login be handling it anymore?..
    - imap-login: Authenticate PLAIN failed: Authentication failed:
      Authentication server isn't connected, try again later.. []
        ^ NO Authentication failed. (should be Temporary login failure!)
    - if auth process dies, login process should retry authentication if
      possible. or if not, disconnect the client so it doesn't think the auth

 - auth
    - with blocking passdb we're not caching lookups if the password was wrong
    - non-plaintext authentication doesn't support all features:
        - multiple passdbs don't work, only the first one is used
	- auth cache's last_success password change check doesn't exist
	- auth_cache_negative_ttl doesn't check password mismatches
    - SIGHUP restarts auth processes .. but does it wait until they've finished
      with all requests? no.
    - does dovecot-auth really break when it runs out of fds?
    - dovecot-auth should limit how fast authentication requests are allowed
      from login processes. especially if there's one login/connection the speed
      should be something like once/sec. also limit how fast to accept new
    - support read-only logins. user could with alternative password get only
      read-access to mails so mails could be read relatively safely with
      untrusted computers. Maybe always send [ALERT] about the previous
      read-only login time with IP?
    - dovecot-auth workers: create a separate dovecot-pam worker which shares
      pretty much all code with dovecot-auth worker but isn't linked against
      any libraries. or..? this might be difficult to do, especially because the
      workers currently can handle any kind of passdb/userb requests. perhaps
      there should be a completely separate simple PAM authenticator binary.

 - master
    - configurable syslog prefix
    - if there are duplicate settings, complain about it

 - quota
    - if dovecot-uidlist can't be written, assume the new mails have UIDs 
      beginning from uidlist.next_uid. Whenever mails are expunged, overwrite
      the next_uid field with the current highest next_uid. Whenever we have
      assumed UIDs and uidlist gets updated, throw the client out with
      "inconsist mailbox".

 - ssl
    - add setting: ssl_options = bitmask. by default we enable all openssl
      workarounds, this could be used to disable some of them
    - gnutls support isn't working

 - search
    - message header search: we should ignore LWSP between two MIME blocks(?)
    - message_search_init() could accept multiple search keywords so we
      wouldn't need to call it separately for each one (so we wouldn't need
      to parse the message multiple times).
    - could optionally support scanning inside file attachments and use
      plugins to extract text out of them (word, excel, pdf, etc. etc.)
    - Create our own extension: When searching with TEXT/BODY, return
      the message text surrounding the keywords just like web search engines
      do. like: SEARCH X-PRINT-MATCHES TEXT "hello" -> * SEARCH 1 "He said:
      Hello world!" 2 "Hello, I'm ...". This would be especially useful with
      the above attachment scanning.

 - general
    - stop using atol(), atoi(), strtoul() etc. in places where we actually
      care about what they return, and rather create our own function which
      checks if the input overflows the integer, and if so call i_fatal()
    - LMTP server
    - ability to build plugins statically into the binaries
    - ~/.dovecotrc to override system wide settings. namespace settings should
      override all the previous namespace settings instead of adding new.
    - things break if next_uid gets to 2^32

 - THREAD=ORDEREDSUBJECT - although pretty useless I'd think.
 - mailbox-referrals (rfc2193)
     - this is useful whenever we would otherwise need to make the
       connection ourself. for example load balancing and shared mailboxes
       requiring another UID to run.
     - this rfc defines no exact way for server to detect if client
       supports referrals or not. I don't think there's much point in
       supporting only referrals, as most clients don't support them.
       Instead we should return referrals when we know that client
       supports them, otherwise do the connecting ourself. If client
       issues RLIST or RLSUB command, it's safe to assume it supports
     - for load balancing this works just fine, but what about shared
       mailboxes which require different UID? If we login with our own
       username, we end up with our own UID instead of what we wanted.
       IMAP URLs don't support separated authorization id which would
       have made this very easy.. We could give the "userid@group" as
       userid, but clients probably treat it as different userid and
       ask the password again.
     - problems, problems, .. maybe not worth the trouble.
 - drafts:
     - annotate (draft-ietf-imapext-annotate)
	 - per-message annotations. this will be major change. especially
	   because currently there's no suitable storage for them, and
	   they'll probably change all the time.. maybe if we moved into
	   berkeley db to store the .data file and these annotations.
	 - this is separate problem from index files. indexes are treated as
	   temporary files, annotations are permanent data. we'd have to
	   support non-db way to do this too, which would probably be just a
	   simple (slow) text file.
	 - use lib-dict probably
     - metadata (draft-daboo-imap-annotatemore)
	 - server and per-mailbox annotations. much easier than
	   per-message annotations, but they'd be easier to place into
	   db as well.
	 - lib-dict, again
     - binary (draft-nerenberg-imap-binary)