view TODO @ 2820:f3e7ce34721f HEAD

config.rpath isn't supposed to be here..
author Timo Sirainen <tss@iki.fi>
date Mon, 25 Oct 2004 21:50:04 +0300
parents 689e4235681f
children 12fc3c22ffe8
line wrap: on
line source

 - lock_method=dotlock, mmap_disable=no crashes in cache locking
 - transaction log: when replacing log with a same sequence, we remove it from
   log's file list, but we don't do anything to existing log views. this can
   crash later in mail_transaction_log_view_set() because 'first' is from log
   list, while we're comparing it into view->tail which it never is. also
   overwriting it leaks memory..
 - mbox: when we're updating flags with lazy writing, we're still parsing the
   mbox, just not writing to it!
 - mbox: always add empty line. make the parser require it too? syncing should
   make sure there always exists two LFs at end of file. raw-mbox-stream should
   make sure the last message ends with LF even if it doesn't exist in the file

	if (sync_ctx.seen_first_mail &&
	    sync_ctx.base_uid_last != sync_ctx.next_uid-1 &&
	    ret == 0 && !sync_ctx.delay_writes) {
     ^ doesn't work correctly with partial syncing?
 - mail_index_lookup_ext() doesn't handle resizes between views. probably
   needs size_t *size_r parameter added..
 - get_field() doesn't work if it's not already cached
 - we probably shouldn't do duplicate detection/fixing?.. or at least stat()
   the old file before trying, because we might have just previously seen the
   old file and then new file and then we try to fix it..

 - mbox: how well does dirty sync + status work? it reads the last mail every
   time? not very good..
 - rawlog process gets stuck when imap dies:
     send(1, 0xFFBFF4C8, 1024, 0)    (sleeping...)
 - subscribe: IMAP(anonymous): open(anonymous/mail/.temp...) failed: Permission denied
 - we can mail_cache_add() same header multiple times..
 - configurable syslog prefix
 - SIGHUP restarts auth processes .. but does it wait until they've finished
   with all requests?
 - SIGHUP rather shouldn't restart listening sockets if they didn't change..
 - dovecot-auth + ldap leaks?
 - subscriptions file should contain namespace prefixes. at least optionally.
   there's the subscriptions = yes setting now for namespaces.. do it so that
   if prefix = "" has subscriptions, it contains prefixes. otherwise not.
 - if auth process dies, login process should retry authentication if possible.
   or if not, disconnect the client so it doesn't think the auth failed.
 - mbox: syncing existing indexes takes 4x longer than creating new one, why?

 - caching
     - force bits should be used only for nonregistered fields
     - change envelope parsing not to use get_headers() so imap.envelope can
       actually be cached without all the headers..
     - compression should drop fields with last_used <
       (latest_mail_index_date - month)
     - when parsing mbox or saving message, parse the mail through index-mail
       so things gets saved into cache immediately

 - maildir: hardlink copying doesn't update indexes

 - dotlocking: stale_timeout should probably take into account how old the
   file is.. so uses more time with more recent locks..

    - COPY doesn't work to itself (lock assert crash)
    - Create UW-IMAP like "DON'T DELETE THIS MESSAGE" message when mbox
      gets empty
    - keep mbox lock for two extra seconds after sync
    - move /var/mail/user to ~/mbox if ~/mbox exists.. supposedly this
      could be useful if /var/mail doesn't have quota, but ~/mail does.
      now, what do we then do if we can move only some of the mails?..
    - if we can't create dotlock file for mbox, make sure it still can be
      selected in read-only state

 - index
    - optimize initial left_idx in mail_index_lookup_uid_range()
    - when referencing log files, open them immediately?
    - if log file is lost, generate it from old and new index
    - make sure when changing flags multiple times in transaction it goes ok.
      especially replace + dirty flag

 - post-login-sql-command
 - add setting: ssl_options = bitmask. by default we enable all openssl
   workarounds, this could be used to disable some of them

 - open("/var/run/dovecot//ssl-parameters.dat", O_RDONLY|O_LARGEFILE) =-1 ENOENT
    ^ loops forever

 - don't return \* in PERMANENTFLAGS when we're full

 - support read-only logins. user could with alternative password get only
   read-access to mails so mails could be read relatively safely with
   untrusted computers. Maybe always send [ALERT] about the previous
   read-only login time with IP?
 - ~/.dovecotrc to override system wide settings. namespace settings should
   override all the previous namespace settings instead of adding new.

 - fix cygwin compile? _close and _read in *stream*.c fails.
 - handle out of quota conditions:
     - if dovecot-uidlist can't be written, assume the new mails have UIDs 
       beginning from uidlist.next_uid. Whenever mails are expunged, overwrite
       the next_uid field with the current highest next_uid. Whenever we have
       assumed UIDs and uidlist gets updated, throw the client out with
       "inconsist mailbox".
     - make sure all syscalls check for ENOSPACE (and ENOACCESS while at it)
 - NFS safety:
    - .customflags: use rename(), but there's a problem when we have to remove
      unused flags to make room for new ones. to fix that add new field in
      the file, it would be set for flags which are currently unused. if that
      field is set, .customflags must be locked before the flag is set to any
      messages. but make sure there's no race conditions, we probably have to
      wait a few seconds just to make sure no-one set a flag we want to remove
    - ESTALE

 - send client IP immediately after accept() to master process. make sure
   master shows the IP if login dies unexpectedly. master should probably also
   kill the login process if it doesn't kill itself soon enough.. or maybe just
   log the IP immediately.

 - ioloop-kqueue.c patch
 - does dovecot-auth really break when it runs out of fds?

 - should we support some non-tokenizing way to parse mail addresses? ..
   eg. "foo ? bar <x@y>" would now show up as "foo? bar <x@y>"

 - when mailbox is deleted/renamed and someone else had it open, we get
   stat() error messages in log file.

 - bugs
    - CREATE a, CREATE b, save mails into them, DELETE a, RENAME b a.
      -> breaks if a+b have same UIDVALIDITY. We could update the
      UIDVALIDITY for the renamed mailbox and all mailboxes under it.
      Then return with tagged "OK [NEW-UIDVALIDITY 1234] Renamed". Assuming
      other IMAP people agree to that.

 - reliability fixes:
    - limit folder hierarchy levels? user can now create eg. a/a/a/a/...
      and then start renaming them from end to beginning, which probably will
      at some point start causing syscall failures which will fill up logs.
    - dotlock overriding is racy, but it's pretty difficult to fix it. Also
      overriding someone else's dotlock in shared folder isn't possible. These
      could be fixed by having separate lock process running as root, which
      would chown() the file for another uid and then unlink() it as that user.
      One problem with that is that if malicious user sets setuid+execute bits
      on for the file, he could run the file and get changed to the new uid.
      That hopefully shouldn't matter much since the new uid should be user
      with minimum possible privileges. Anyway, optional..

 - checks:
   - make sure SELECT rebuilds index properly when next_uid is near 32bit value
   - make sure connection limits work
   - check if some asserts should be changed into if/i_panic code, so that
     disabling asserts would be possible

 - enhancements:
    - option to disable SORT, SEARCH and other memory/cpu-intensive features.
      defaults and per-user by dovecot-auth.
    - maildir could support also the dirty-flag in messages. files would be
      renamed "whenever there's time" (that'd require the indexer program, or
      forking and doing it in background)
    - OpenSSL: support generated DH parameters
    - SSL: Support password protected key files. Support reading the password
      from user at runtime (dovecot startssl or something). currently it just
      hangs.
    - check with strace what dovecot does when evolution checks new mail,
      it's quite a lot. some things probably wouldn't need to be done
      (mkdirs inbox) and other things could be cached in memory.
    - sort: we could create alternative indexes for different sort conditions.
      sort code itself already supports this optimization.
    - support zlib compressed mbox/maildir? mbox maybe just read-only.
      do it through istream-zlib wrapper
    - THREAD=ORDEREDSUBJECT - although pretty useless I'd think.

 - logging
    - Login: username 1.2.3.4:1025 5.6.7.8:993 imaps,compressed
    - Logout: username 1.2.3.4:1025 5.6.7.8:993 imaps,compressed in:1000 out:1000000
    - n failed login attepts (before failure or success, once in n seconds)

 - lib-charset:
     - utf8_toupper() is a must. and a bit difficult if we want to do it right.
     - add support for other things than iconv() as well? we could reuse
       the code from cyrus or courier
     - cache iconvs? they'd probably be faster if we just reset the
       conversion instead of opening new one every time. and there will likely
       be only one or two charsets which are used for nearly all conversions.

 - should we allow following symlinks in mbox/maildirs? they are now.
    - if we implement shared mailboxes with shared indexes, never do that or
      others could symlink your personal mailboxes and see the indexes
      created for it which may contain envelope etc. data
    - this allows circular mailbox hierarchies which should be prevented by
      eg. allowing max. 20 hierarchies.

 - index:
    - we could try compressing same from/to/subject fields into a single
      location in data file. requires larger changes..
    - Most messages are text/plain/7bit/us-ascii/no-other-content-type-params.
      Instead of saving tons of nearly identical BODY/BODYSTRUCTURE fields
      into cache, we could just set INDEX_MAIL_FLAG_TEXT_PLAIN_7BIT_ASCII bit
      on and generate the bodystructure for such messages on the fly.
    - read-only support for mailboxes where we don't have write-access
    - if .customflags is removed and Maildir files have custom flags, add
      "unknown1" "unknown2" etc. flags to .customflags file for each found flag
    - support storing message headers into indexes. this could be useful when
      indexes are in local disk but actual mails are accessed through NFS.
    - if we wanted to support huge mailboxes with small memory usage, it'd now
      be possible if we just instead of mmap()ing the whole index files would
      have maybe 3-4 256k mmap()ed areas which we move based on the need.
	- should work fine with imap.index and imap.index.cache
	- log files aren't affected by mailbox size
	- but, is it worth the trouble really? the OS can do all this itself,
	  only thing we're doing is keeping the processes virtual memory usage
	  small.

 - SEARCH:
    - message_body_search() could accept multiple search keywords so we
      wouldn't need to call it separately for each one (so we wouldn't need
      to parse the message multiple times).
    - message_body_search() could support NULL MessagePart and the searching
      could be done while parsing the message. this would need changes to
      message_parse() as well.
    - could optionally support scanning inside file attachments and use
      plugins to extract text out of them (word, excel, pdf, etc. etc.)
    - use a trie index for fast text searching, like cyrus squat?
    - Create our own extension: When searching with TEXT/BODY, return
      the message text surrounding the keywords just like web search engines
      do. like: SEARCH X-PRINT-MATCHES TEXT "hello" -> * SEARCH 1 "He said:
      Hello world!" 2 "Hello, I'm ...". This would be especially useful with
      the above attachment scanning.

 - general:
    - sieve (rfc3028), we can use Cyrus Sieve
    - rfc2231 continuation support
    - rfc2557 support for BODYSTRUCTURE, as specified by RFC3501
    - lmtp server - is it needed? dovecot-deliver binary at least would be
      useful
    - create indexer binary
    - support Maildir++ quota

 - cleanups:
    - check if t_push()/t_pop() should be added somewhere
    - try to fix @UNSAFE code to use buffer API instead
	- subscription-file.c, custom_flags
	- [io]stream-file.c?
    - grep for FIXME

 - auth / login:
    - kchuid, SRP
    - Digest-MD5: support integrity protection, and maybe crypting. Do it
      through login process like SSL is done?
    - dovecot-auth should limit how fast authentication requests are allowed from
      login processes. especially if there's one login/connection the speed
      should be something like once/sec. also limit how fast to accept new
      connections.
    - if passdb = userdb, ask them both in one query and cache it. implement
      passwordcheck userdb using the caching.

capabilities:
 - preferrably all should be possible to #ifdef away by a configure
   option (--without-capabilities=acl,namespace,...)
 - possibility to disable them from config file
 - acl (rfc2086, draft-ietf-imapext-acl), namespace (rfc2342)
     - probably do it like cyrus. "user.<username>" to access other
       users, with "" defaulting to "user.<myself>". these should be
       configurable however.
     - shared namespaces? maybe configurable in config file
     - easiest way to do ACL would be to use unix modes, but is that
       useful at all? Well, ACL2 has a bit better support for that, so
       maybe we could support it.
     - otherwise gets a bit trickly, we could keep all mail in "imapmail"
       group and 0600/0700 mode by default, but when mail is shared to others,
       the group read/write access bits would be set. or alternatively we
       could launch another imap process to handle it, which we should support
       anyway. ACLs could be stored into ".acl" ascii file in each folder.
     - support for private and shared flags, configurable by mailbox admin.
       this isn't in any draft yet, but ACL2 author was going to create one.
       [SHAREDFLAGS (...)] would specify which ones are shared, don't know yet
       how they would be configured.
 - quota (rfc2087, draft-cridland-imap-quota)
     - give filesystem values only to admins
     - support for Maildir++, probably no need to support more.
       quota capability supports complex quota configuration, but if
       no mailer supports them we probably shouldn't bother either
 - id (rfc2971)
     - must be configurable what gets sent, default to only name=Dovecot
     - separate pre/post-login settings
     - optionally log configured parts of the client information, but only
       once, probably at the same time as logging "Logged in",
       "Disconnected", etc.
     - remember to force truncating values longer than 30 chars,
       especially before logging
 - mailbox-referrals (rfc2193)
     - this is useful whenever we would otherwise need to make the
       connection ourself. for example load balancing and shared mailboxes
       requiring another UID to run.
     - this rfc defines no exact way for server to detect if client
       supports referrals or not. I don't think there's much point in
       supporting only referrals, as most clients don't support them.
       Instead we should return referrals when we know that client
       supports them, otherwise do the connecting ourself. If client
       issues RLIST or RLSUB command, it's safe to assume it supports
       referrals.
     - for load balancing this works just fine, but what about shared
       mailboxes which require different UID? If we login with our own
       username, we end up with our own UID instead of what we wanted.
       IMAP URLs don't support separated authorization id which would
       have made this very easy.. We could give the "userid@group" as
       userid, but clients probably treat it as different userid and
       ask the password again.
     - problems, problems, .. maybe not worth the trouble.
 - drafts:
     - http://www.imc.org/ids.html
     - annotate (draft-ietf-imapext-annotate)
	 - per-message annotations. this will be major change. especially
	   because currently there's no suitable storage for them, and
	   they'll probably change all the time.. maybe if we moved into
	   berkeley db to store the .data file and these annotations.
	 - this is separate problem from index files. indexes are treated as
	   temporary files, annotations are permanent data. we'd have to
	   support non-db way to do this too, which would probably be just a
	   simple (slow) text file.
     - annotatemore (draft-daboo-imap-annotatemore)
	 - server and per-mailbox annotations. much easier than
	   per-message annotations, but they'd be easier to place into
	   db as well.
     - binary (draft-nerenberg-imap-binary)
	 - perhaps not too useful. I'd like to make Dovecot fully
	   binary-safe though.
     - view (draft-ietf-imapext-view)
         - slow, complex, luckily draft expired almost two years ago.
	   i hope i don't have to implement this :)
	 - can be done client-side just fine (evolution's virtual folders)