Mercurial > dovecot > original-hg > dovecot-1.2
view TODO @ 973:d634f81c1217 HEAD
Don't call hash_remove() if it isn't there.
author | Timo Sirainen <tss@iki.fi> |
---|---|
date | Tue, 14 Jan 2003 13:43:01 +0200 |
parents | 6f005d5d9931 |
children | 7bd8508ed0fa |
line wrap: on
line source
- bugs - maildir: if mail file isn't found, it may be because it was renamed (flag changed). we must then sync the directory and see again if the mail is found - we may override mbox dotlock file, but then get stuck at fcntl/flock. we should check for those before overriding the mbox lock.. - mail-lockdir.c isn't 100% safe.. stale locks are detected by checking that hard link count is 1, then it's unlink()ed. but what if another process did the same unlink() + creat() in the middle of our stat()..unlink()? no easy way to fix this really, just replace it with a fcntl() lock. - SEARCH FROM/TO/CC/BCC now generates the field from ENVELOPE which it uses for matching. This however may give different results than when matching from headers. - SIGHUP didn't update imap_listen. this is a bit annoying to fix though, since new listen() may fail for a few times because auth processes may not die immediately.. - SIGHUP doesn't update log file location. - We can use Linux sendfile() up to 2GB, after that we get EOVERFLOW and fail. We should rather fallback to mmap+write at that point. - unlink_directory() is racy with symlink handling, see if that could be helped.. - reliability fixes: - if we deleted mail from index but didn't write modify log, other dovecots don't handle it properly. they either assert at index-sync.c:42 or if new mails have also been added since, they don't notice it at all actually, that breaks reads as well since we get expunges only from the old file.. and check that deleting file does "inconsistency error" - if imap process notices that both modify logs are getting full because it's client isn't syncing, the client should be disconnected - what happens if .customflags can't be locked while opening index? - we don't handle out of memory conditions too well, malloc failing kills the process which is good enough (and likely never happens), but mmap() failures aren't handled too well. Rather should be handled in similiar way to locking failures, so that at least we don't try to rebuild the index because of it. - limit folder hierarchy levels? user can now create eg. a/a/a/a/... and then start renaming them from end to beginning, which probably will at some point start causing syscall failures which will fill up logs. - fsck should check binary tree - checks: - if we have entries in modifylog with UID 10..11, 9..12, 8..13 etc. do they work correctly? - check that search message-id worked properly always - check that search's OR and () work properly - Should SEARCH SENT* apply timezone? - make sure SELECT rebuilds index properly when next_uid is near 32bit value - make sure connection limits work - enhancements: - option to disable SORT, SEARCH and other memory/cpu-intensive features. defaults and per-user by imap-auth. - optionally don't fail if index is locked, but build it in memory - when fetching body/envelope/etc we could try to cache it immediately if we can get lock with try_lock. - optionally use only in-memory indexes - maildir could support also the dirty-flag in messages. files would be renamed "whenever there's time" (that'd require the indexer program, or forking and doing it in background) - optionally keep the message file name as it's UID. Then we don't have to save the filename anywhere. - send EXISTS immediately after new mail arrives. - linux: we can use dnotify for maildir (but not mbox I think, we'd get interrupted all the time if we checked eg. large /var/spool/mail) - *bsd: kqueue() can notify changes in mbox and maildir - .subscriptions currently uses fcntl() locking - maybe we should instead just write to temp file and rename()? optionally at least, so it works with NFS. - OpenSSL: support generated DH parameters - multiline headers can cause our memory usage to go up. that should be fixed somehow. try to change things to be able to handle one line at a time? Well, other IMAP servers have same problem - post 1.0 problem. - check with strace what dovecot does when evolution checks new mail, it's quite a lot. some things probably wouldn't need to be done (mkdirs/symlinking inbox) and other things could be cached in memory. - sort: we could create alternative binary tree file(s) for different sort conditions, ".tree-sort" or something. sort code itself already supports this optimization. - tree file: should we instead use b+-tree or something similiar? or at least try to do some defragmentation with it, so that the root nodes would be kept at the beginning of the file. - use vsftpd-like safebufs, ie. keep non-rwx page before and after the memory we want to use. - mmap_anon() - mmap()ing files would probably need to first go through anon_mmap() and then use MAP_FIXED. annoying that it slows the mmaping.. - data stack should use mmap_anon() - option: copy /var/mail/$user to INBOX when logged in. nice for not missing any mails with quota enabled - support zlib compressed mbox/maildir? mbox maybe just read-only. - THREAD=ORDEREDSUBJECT - although pretty useless I'd think. - logging - Login: username 1.2.3.4:1025 5.6.7.8:993 imaps,compressed - Logout: username 1.2.3.4:1025 5.6.7.8:993 imaps,compressed in:1000 out:1000000 - n failed login attepts (before failure or success, once in n seconds) - lib-charset: - utf8_toupper() is a must. and a bit difficult if we want to do it right. - add support for other things than iconv() as well? we could reuse the code from cyrus or courier - cache iconvs? they'd probably be faster if we just reset the conversion instead of opening new one every time. and there will likely be only one or two charsets which are used for nearly all conversions. - should we allow following symlinks in mbox/maildirs? they are now. - if we implement shared mailboxes with shared indexes, never do that or others could symlink your personal mailboxes and see the indexes created for it which may contain envelope etc. data - this allows circular mailbox hierarchies which should be prevented by eg. allowing max. 20 hierarchies. - allow index files to be in completely separate location than mail data. mails could be read through slow NFS access but indexes from fast local disk. with this thinking it makes more sense to create larger index files to save for example mail headers. also index rebuilding should be very light operation, the indexes would be filled while the data is being accessed by the imap client. of course all this should be optional so we don't slow down when mails and indexes are stored in same disk. - we need permanent storage for UIDs. with mbox use X-UID like UW-IMAP, with maildir a) file:2,flags,Uuid b) file,U=uid:2,flags. uid validity would be in .uidvalidity file. the b-case would require that to be done by the client moving it from new/ to cur/ index: - mbox: - if a file isn't valid mbox and it's tried to be opened, say it in one line in error log, not 6.. - empty lines at beginning of file still aren't ignored - UW-IMAPd writes empty spaces after X-Keywords which it uses so that it doesn't have to rewrite the whole file if status flags changed in the beginning of it. We could do that too. - When expunging the first message we could move the X-IMAPbase header to next message to avoid full rewriting later. - We shouldn't send X-IMAPbase, Status, X-Status, X-Keywords, X-UID, etc. headers to client - they may change and clients must see messages as immutable. - COPY 1 copies X-IMAPbase header too which isn't good idea. save() could actually strip this (and X-UID) while also fixing From-lines etc. - we need either From-line escaping or writing Content-Length when saving mails. - two adjacent From-lines breaks us. not too easy to fix though. - we could try compressing same from/to/subject fields into a single location in data file. requires larger changes.. - read-only support for mailboxes where we don't have write-access - we should try to avoid completely rebuilding indexes unless they're corrupted. especially if we later want to support some read-only boxes and keep the mail flags only in index file. fsck() could verify that records are ok, and that if data file isn't ok the record is deleted. - if .customflags is removed and Maildir files have custom flags, add "unknown1" "unknown2" etc. flags to .customflags file for each found flag - when index is being rebuilt, it always complains about tree/modifylog having wrong indexid.. - if we wanted to support huge mailboxes with small memory usage, it'd now be possible if we just instead of mmap()ing the whole index files would have maybe 3-4 256k mmap()ed areas which we move based on the need. - should work fine with .imap.index and .imap.index.data - log files aren't affected by mailbox size - if the tree file also kept constantly moving the nodes so that tree's root was at the beginning of the file, we could use this mmap caching with it too - but, is it worth the trouble really? the OS can do all this itself, only thing we're doing is keeping the processes virtual memory usage small. lib-storage: - support multiple mailbox formats and locations for one user. that would require support for multiple MailStorages, and since we're chroot()ed, usually the only way to communicate with others would be to create RemoteMailStorage which would use TCP/UNIX sockets to connect to another imap session. - SEARCH: - message_body_search() could accept multiple search keywords so we wouldn't need to call it separately for each one (so we wouldn't need to parse the message multiple times). - message_body_search() could support NULL MessagePart and the searching could be done while parsing the message. this would need changes to message_parse() as well. - could optionally support scanning inside file attachments and use plugins to extract text out of them (word, excel, pdf, etc. etc.) - use a trie index for fast text searching, like cyrus squat? - Create our own extension: When searching with TEXT/BODY, return the message text surrounding the keywords just like web search engines do. like: SEARCH X-PRINT-MATCHES TEXT "hello" -> * SEARCH 1 "He said: Hello world!" 2 "Hello, I'm ...". This would be especially useful with the above attachment scanning. - DELETE/RENAME: when someone else had the mailbox open, we should disconnect it (when stat() fails with ENOENT while syncing). Also deleting selected mailbox begins giving internal error messages. - RENAME INBOX isn't atomic with Maildir. And in general, RENAME can't move mails between different storages. Maybe support doing also using COPY + delete once COPY is atomic? - maildir: atomic COPY could be done by having transaction directories. Make a "tra" directory at the same level as cur/new/tmp, and make it have subdirectories in the same way as tmp has temp files. Directory begins with a "." as long as transaction isn't finished, rename()ing it away finishes it. All mails under finished dirs must be moved into new/ directory and the directory removed by any process who notices them. - we should probably do some light checking that appended mails actually look like valid rfc822 mails.. - maybe limit the length of custom flags? we don't really have a problem with them, but with mbox a long X-IMAPbase could break something.. Maybe configurable, default to 50 chars? - we could send flag changes after all commands by making expunge/flags sync counters separate for modify log. flags would need to update the seq though, too slow? - things calling message_send() could verify that it wrote enough data. if not, fill the rest with spaces and return failure. -1 = error, 0 = filled, 1 = ok. general: - sieve (rfc3028) - rfc2231 continuation support - rfc2557 support for BODYSTRUCTURE, as specified by latest IMAP4rev1 draft - create indexer binary - update docs/index.txt - support Maildir++ quota - maybe give more untagged NO/ALERT replies? like when mailbox is in inconsistent state. and when UIDs are reordered because they're too large. - imap/ and lib-imap/ should allow infinite number of custom flags, it's storage's problem if it can't handle too many of them. auth / login: - kchuid, SRP, anonymous SASL - PAM: support some options so /etc/passwd-lookup isn't needed. uid=x, gid=y, mailroot=/var/mail. maildirs should be then created when needed - Digest-MD5: support integrity protection, and maybe crypting. Do it through imap-login like SSL is done? - for invalid user/pass, wait for a while before giving a reply to user - imap-auth should limit how fast authentication requests are allowed from login processes. especially if there's one login/connection the speed should be something like once/sec. also limit how fast to accept new connections. cleanups / checks: - grep for FIXME - check if t_push()/t_pop() should be added somewhere - try to fix @UNSAFE code to use buffer API instead - subscription-file.c, custom_flags, ioloop - [io]stream-file.c? optional optimizations: - provide some helper binary to save new mail into mailboxes with CR+LF line breaks? - disk I/O is the biggest problem, so split the mail into multiple computers based on user and have a proxy in the front redirecting the connection. cyrus had something like this except a lot more complicated - it tried to fix the problem of having shared mailboxes. we have the same problem with local shared mailboxes as we don't use same UID for everyone's mail and we may be chrooted, so locally we could communicate with UNIX sockets, remotely that could be done with TCP sockets. capabilities: - preferrably all should be possible to #ifdef away by a configure option (--without-capabilities=acl,namespace,...) - possibility to disable them from config file - acl (rfc2086, draft-ietf-imapext-acl), namespace (rfc2342) - probably do it like cyrus. "user.<username>" to access other users, with "" defaulting to "user.<myself>". these should be configurable however. - shared namespaces? maybe configurable in config file - easiest way to do ACL would be to use unix modes, but is that useful at all? Well, ACL2 has a bit better support for that, so maybe we could support it. - otherwise gets a bit trickly, we could keep all mail in "imapmail" group and 0600/0700 mode by default, but when mail is shared to others, the group read/write access bits would be set. or alternatively we could launch another imap process to handle it, which we should support anyway. ACLs could be stored into ".acl" ascii file in each folder. - support for private and shared flags, configurable by mailbox admin. this isn't in any draft yet, but ACL2 author was going to create one. [SHAREDFLAGS (...)] would specify which ones are shared, don't know yet how they would be configured. - quota (rfc2087, draft-cridland-imap-quota) - give filesystem values only to admins - support for Maildir++, probably no need to support more. quota capability supports complex quota configuration, but if no mailer supports them we probably shouldn't bother either - id (rfc2971) - must be configurable what gets sent, default to only name=Dovecot - separate pre/post-login settings - optionally log configured parts of the client information, but only once, probably at the same time as logging "Logged in", "Disconnected", etc. - remember to force truncating values longer than 30 chars, especially before logging - mailbox-referrals (rfc2193) - this is useful whenever we would otherwise need to make the connection ourself. for example load balancing and shared mailboxes requiring another UID to run. - this rfc defines no exact way for server to detect if client supports referrals or not. I don't think there's much point in supporting only referrals, as most clients don't support them. Instead we should return referrals when we know that client supports them, otherwise do the connecting ourself. If client issues RLIST or RLSUB command, it's safe to assume it supports referrals. - for load balancing this works just fine, but what about shared mailboxes which require different UID? If we login with our own username, we end up with our own UID instead of what we wanted. IMAP URLs don't support separated authorization id which would have made this very easy.. We could give the "userid@group" as userid, but clients probably treat it as different userid and ask the password again. - problems, problems, .. maybe not worth the trouble. - literal+ (rfc2088) - simple. in case of invalid data, just disconnect client. - idle (rfc2177) - just call the syncing every few seconds (configurable) - with Linux we can use fcntl() and F_SETSIG to provide fast checks. just make sure sync() still won't be called more than once in a few seconds - uidplus (rfc2359) - uid expunge: no problem - append, copy: oh no. these would slow down things and make handling them much more difficult. currently we just store the mails to destination mailbox without touching the indexes. since we'd need to know their final UID, we'd have to lock the indexes and mbox) fsck() first and append() next to find out the uid, maildir) move the mail directly into cur/ and index it. - unselect (no draft or anything AFAIK) - like CLOSE, but doesn't expunge mails. easy. - drafts: - http://www.imc.org/ids.html - multiappend (draft-crispin-imap-multiappend) - shouldn't have any problems - listext (draft-ietf-imapext-list-extensions) - well, it expired January 2002.. I like it though. - children (draft-gahrns-imap-child-mailbox) - I like listext more.. They have the same functionality though, so pretty easy to support both if needed - annotate (draft-ietf-imapext-annotate) - per-message annotations. this will be major change. especially because currently there's no suitable storage for them, and they'll probably change all the time.. maybe if we moved into berkeley db to store the .data file and these annotations. - annotatemore (draft-daboo-imap-annotatemore) - server and per-mailbox annotations. much easier than per-message annotations, but they'd be easier to place into db as well. - binary (draft-nerenberg-imap-binary) - perhaps not too useful. I'd like to make Dovecot fully binary-safe though. - view (draft-ietf-imapext-view) - slow, complex, luckily draft expired almost two years ago. i hope i don't have to implement this :) - can be done client-side just fine (evolution's virtual folders)