Mercurial > dovecot > core-2.2

annotate src/lib/istream-mmap.c @ 22664:fea53c2725c0

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
director: Fix director_max_parallel_moves/kicks type Should be uint, not time.
author Timo Sirainen <timo.sirainen@dovecot.fi>
date Thu, 09 Nov 2017 12:24:16 +0200
parents 2e2563132d5f
children cb108f786fb4
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
21390
2e2563132d5f Updated copyright notices to include the year 2017.
Stephan Bosch <stephan.bosch@dovecot.fi>
parents: 21365
diff changeset
1 /* Copyright (c) 2002-2017 Dovecot authors, see the included COPYING file */
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
2
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
3 #include "lib.h"
3241
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
4 #include "ioloop.h"
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
5 #include "mmap-util.h"
13529
cf77e448295c Renamed lib/*-internal.h files to lib/*-private.h for consistency.
Timo Sirainen <tss@iki.fi>
parents: 12782
diff changeset
6 #include "istream-private.h"
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
7
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
8 #include <unistd.h>
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
9 #include <sys/stat.h>
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
10
903
fd8888f6f037 Naming style changes, finally got tired of most of the typedefs. Also the
Timo Sirainen <tss@iki.fi>
parents: 764
diff changeset
11 struct mmap_istream {
6415
b0096861c390 Renamed struct _[io]stream to struct [io]stream_private. Also removed _
Timo Sirainen <tss@iki.fi>
parents: 6411
diff changeset
12 struct istream_private istream;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
13
3241
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
14 struct timeval fstat_cache_stamp;
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
15
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
16 void *mmap_base;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
17 off_t mmap_offset;
1870
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
18 uoff_t v_size;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
19
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
20 unsigned int autoclose_fd:1;
903
fd8888f6f037 Naming style changes, finally got tired of most of the typedefs. Also the
Timo Sirainen <tss@iki.fi>
parents: 764
diff changeset
21 };
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
22
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
23 static size_t mmap_pagemask = 0;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
24
16020
6cabb95d32ec iostreams: Added close_parent flag to close() handler and clarified close/destroy APIs.
Timo Sirainen <tss@iki.fi>
parents: 15715
diff changeset
25 static void i_stream_mmap_close(struct iostream_private *stream,
6cabb95d32ec iostreams: Added close_parent flag to close() handler and clarified close/destroy APIs.
Timo Sirainen <tss@iki.fi>
parents: 15715
diff changeset
26 bool close_parent ATTR_UNUSED)
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
27 {
903
fd8888f6f037 Naming style changes, finally got tired of most of the typedefs. Also the
Timo Sirainen <tss@iki.fi>
parents: 764
diff changeset
28 struct mmap_istream *mstream = (struct mmap_istream *) stream;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
29
3241
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
30 if (mstream->autoclose_fd && mstream->istream.fd != -1) {
10847
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
31 if (close(mstream->istream.fd) < 0) {
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
32 i_error("mmap_istream.close(%s) failed: %m",
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
33 i_stream_get_name(&mstream->istream.istream));
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
34 }
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
35 }
4766
b96086a311a9 When i/ostream is closed, change the fd to be -1 so it won't be accidentally
Timo Sirainen <tss@iki.fi>
parents: 3863
diff changeset
36 mstream->istream.fd = -1;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
37 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
38
903
fd8888f6f037 Naming style changes, finally got tired of most of the typedefs. Also the
Timo Sirainen <tss@iki.fi>
parents: 764
diff changeset
39 static void i_stream_munmap(struct mmap_istream *mstream)
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
40 {
6415
b0096861c390 Renamed struct _[io]stream to struct [io]stream_private. Also removed _
Timo Sirainen <tss@iki.fi>
parents: 6411
diff changeset
41 struct istream_private *_stream = &mstream->istream;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
42
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
43 if (_stream->buffer != NULL) {
10847
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
44 if (munmap(mstream->mmap_base, _stream->buffer_size) < 0) {
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
45 i_error("mmap_istream.munmap(%s) failed: %m",
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
46 i_stream_get_name(&_stream->istream));
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
47 }
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
48 mstream->mmap_base = NULL;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
49 _stream->buffer = NULL;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
50 _stream->buffer_size = 0;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
51 mstream->mmap_offset = 0;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
52 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
53 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
54
6420
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
55 static void i_stream_mmap_destroy(struct iostream_private *stream)
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
56 {
903
fd8888f6f037 Naming style changes, finally got tired of most of the typedefs. Also the
Timo Sirainen <tss@iki.fi>
parents: 764
diff changeset
57 struct mmap_istream *mstream = (struct mmap_istream *) stream;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
58
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
59 i_stream_munmap(mstream);
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
60 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
61
6415
b0096861c390 Renamed struct _[io]stream to struct [io]stream_private. Also removed _
Timo Sirainen <tss@iki.fi>
parents: 6411
diff changeset
62 static size_t mstream_get_mmap_block_size(struct istream_private *stream)
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
63 {
5479
b3999af65adb If istream implementation doesn't set set_max_buffer_size() or stat(),
Timo Sirainen <tss@iki.fi>
parents: 5106
diff changeset
64 return (stream->max_buffer_size + mmap_get_page_size() - 1) & ~
b3999af65adb If istream implementation doesn't set set_max_buffer_size() or stat(),
Timo Sirainen <tss@iki.fi>
parents: 5106
diff changeset
65 (mmap_get_page_size() - 1);
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
66 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
67
6420
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
68 static ssize_t i_stream_mmap_read(struct istream_private *stream)
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
69 {
903
fd8888f6f037 Naming style changes, finally got tired of most of the typedefs. Also the
Timo Sirainen <tss@iki.fi>
parents: 764
diff changeset
70 struct mmap_istream *mstream = (struct mmap_istream *) stream;
1870
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
71 size_t aligned_skip;
1146
ee4bdf40ec10 Bugfixes to handling >2GB files.
Timo Sirainen <tss@iki.fi>
parents: 953
diff changeset
72 uoff_t top;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
73
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
74 if (stream->pos < stream->buffer_size) {
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
75 /* more bytes available without needing to mmap() */
1870
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
76 stream->pos = stream->buffer_size;
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
77 return stream->pos - stream->skip;
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
78 }
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
79
2431
8e9d8cb1e674 Removed istream.disconnected, it's too much of a special case and the only
Timo Sirainen <tss@iki.fi>
parents: 2421
diff changeset
80 if (stream->istream.v_offset >= mstream->v_size) {
8e9d8cb1e674 Removed istream.disconnected, it's too much of a special case and the only
Timo Sirainen <tss@iki.fi>
parents: 2421
diff changeset
81 stream->istream.eof = TRUE;
1870
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
82 return -1;
2431
8e9d8cb1e674 Removed istream.disconnected, it's too much of a special case and the only
Timo Sirainen <tss@iki.fi>
parents: 2421
diff changeset
83 }
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
84
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
85 aligned_skip = stream->skip & ~mmap_pagemask;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
86 if (aligned_skip == 0 && mstream->mmap_base != NULL) {
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
87 /* didn't skip enough bytes */
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
88 return -2;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
89 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
90
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
91 stream->skip -= aligned_skip;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
92 mstream->mmap_offset += aligned_skip;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
93
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
94 if (mstream->mmap_base != NULL) {
10847
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
95 if (munmap(mstream->mmap_base, stream->buffer_size) < 0) {
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
96 i_error("mmap_istream.munmap(%s) failed: %m",
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
97 i_stream_get_name(&stream->istream));
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
98 }
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
99 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
100
1883
aa2341461620 cleanups
Timo Sirainen <tss@iki.fi>
parents: 1870
diff changeset
101 top = mstream->v_size - mstream->mmap_offset;
5479
b3999af65adb If istream implementation doesn't set set_max_buffer_size() or stat(),
Timo Sirainen <tss@iki.fi>
parents: 5106
diff changeset
102 stream->buffer_size = I_MIN(top, mstream_get_mmap_block_size(stream));
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
103
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
104 i_assert((uoff_t)mstream->mmap_offset + stream->buffer_size <=
1883
aa2341461620 cleanups
Timo Sirainen <tss@iki.fi>
parents: 1870
diff changeset
105 mstream->v_size);
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
106
2182
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
107 if (stream->buffer_size == 0) {
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
108 /* don't bother even trying mmap */
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
109 mstream->mmap_base = NULL;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
110 stream->buffer = NULL;
2182
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
111 } else {
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
112 mstream->mmap_base =
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
113 mmap(NULL, stream->buffer_size, PROT_READ, MAP_PRIVATE,
3241
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
114 stream->fd, mstream->mmap_offset);
2182
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
115 if (mstream->mmap_base == MAP_FAILED) {
8196
40a07553606c Input streams: Improved error handling and added more asserts.
Timo Sirainen <tss@iki.fi>
parents: 7086
diff changeset
116 i_assert(errno != 0);
2182
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
117 stream->istream.stream_errno = errno;
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
118 mstream->mmap_base = NULL;
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
119 stream->buffer = NULL;
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
120 stream->buffer_size = 0;
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
121 stream->skip = stream->pos = 0;
16773
76d5e3c8cec3 iostreams: Set stream error string when it provides extra information.
Timo Sirainen <tss@iki.fi>
parents: 16020
diff changeset
122 io_stream_set_error(&stream->iostream,
76d5e3c8cec3 iostreams: Set stream error string when it provides extra information.
Timo Sirainen <tss@iki.fi>
parents: 16020
diff changeset
123 "mmap() failed: %m");
10847
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
124 i_error("mmap_istream.mmap(%s) failed: %m",
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
125 i_stream_get_name(&stream->istream));
2182
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
126 return -1;
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
127 }
4e559fad61fb minor optimization
Timo Sirainen <tss@iki.fi>
parents: 1915
diff changeset
128 stream->buffer = mstream->mmap_base;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
129 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
130
2439
7d6ca38c8c37 Added mmap_get_page_size() call which basically returns getpagesize() but
Timo Sirainen <tss@iki.fi>
parents: 2431
diff changeset
131 if (stream->buffer_size > mmap_get_page_size()) {
1870
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
132 if (madvise(mstream->mmap_base, stream->buffer_size,
10847
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
133 MADV_SEQUENTIAL) < 0) {
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
134 i_error("mmap_istream.madvise(%s): %m",
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
135 i_stream_get_name(&stream->istream));
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
136 }
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
137 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
138
1870
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
139 stream->pos = stream->buffer_size;
8196
40a07553606c Input streams: Improved error handling and added more asserts.
Timo Sirainen <tss@iki.fi>
parents: 7086
diff changeset
140 i_assert(stream->pos - stream->skip > 0);
1870
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
141 return stream->pos - stream->skip;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
142 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
143
6420
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
144 static void i_stream_mmap_seek(struct istream_private *stream, uoff_t v_offset,
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
145 bool mark ATTR_UNUSED)
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
146 {
903
fd8888f6f037 Naming style changes, finally got tired of most of the typedefs. Also the
Timo Sirainen <tss@iki.fi>
parents: 764
diff changeset
147 struct mmap_istream *mstream = (struct mmap_istream *) stream;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
148
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
149 if (stream->buffer_size != 0 &&
1883
aa2341461620 cleanups
Timo Sirainen <tss@iki.fi>
parents: 1870
diff changeset
150 (uoff_t)mstream->mmap_offset <= v_offset &&
aa2341461620 cleanups
Timo Sirainen <tss@iki.fi>
parents: 1870
diff changeset
151 (uoff_t)mstream->mmap_offset + stream->buffer_size > v_offset) {
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
152 /* already mmaped */
1883
aa2341461620 cleanups
Timo Sirainen <tss@iki.fi>
parents: 1870
diff changeset
153 stream->skip = stream->pos = v_offset - mstream->mmap_offset;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
154 } else {
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
155 /* force reading next time */
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
156 i_stream_munmap(mstream);
1883
aa2341461620 cleanups
Timo Sirainen <tss@iki.fi>
parents: 1870
diff changeset
157 stream->skip = stream->pos = v_offset;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
158 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
159
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
160 stream->istream.v_offset = v_offset;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
161 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
162
6420
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
163 static void i_stream_mmap_sync(struct istream_private *stream)
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
164 {
1870
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
165 struct mmap_istream *mstream = (struct mmap_istream *) stream;
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
166
3241
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
167 i_stream_munmap(mstream);
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
168 stream->skip = stream->pos = stream->istream.v_offset;
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
169
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
170 mstream->fstat_cache_stamp.tv_sec = 0;
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
171 }
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
172
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
173 static int fstat_cached(struct mmap_istream *mstream)
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
174 {
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
175 if (mstream->fstat_cache_stamp.tv_sec == ioloop_timeval.tv_sec &&
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
176 mstream->fstat_cache_stamp.tv_usec == ioloop_timeval.tv_usec)
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
177 return 0;
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
178
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
179 if (fstat(mstream->istream.fd, &mstream->istream.statbuf) < 0) {
10847
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
180 i_error("mmap_istream.fstat(%s) failed: %m",
5f16e488e7f6 i/ostreams can now have a name (e.g. file path).
Timo Sirainen <tss@iki.fi>
parents: 10582
diff changeset
181 i_stream_get_name(&mstream->istream.istream));
21365
cc6e4b239003 istream-mmap: Mark stream eof on error and copy errno
Aki Tuomi <aki.tuomi@dovecot.fi>
parents: 19552
diff changeset
182 mstream->istream.istream.stream_errno = errno;
3241
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
183 return -1;
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
184 }
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
185
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
186 mstream->fstat_cache_stamp = ioloop_timeval;
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
187 return 0;
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
188 }
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
189
14964
6d2be8d8891c i_stream_stat() API changed.
Timo Sirainen <tss@iki.fi>
parents: 14133
diff changeset
190 static int
6420
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
191 i_stream_mmap_stat(struct istream_private *stream, bool exact ATTR_UNUSED)
3241
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
192 {
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
193 struct mmap_istream *mstream = (struct mmap_istream *) stream;
b79853b4b005 Replaced i_stream_get_size() with i_stream_stat(). Added i_stream_sync().
Timo Sirainen <tss@iki.fi>
parents: 3239
diff changeset
194
14964
6d2be8d8891c i_stream_stat() API changed.
Timo Sirainen <tss@iki.fi>
parents: 14133
diff changeset
195 return fstat_cached(mstream);
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
196 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
197
6142
6c0bfc35af03 Removed memory pool parameter from iostreams. Default pool was almost always
Timo Sirainen <tss@iki.fi>
parents: 5479
diff changeset
198 struct istream *i_stream_create_mmap(int fd, size_t block_size,
903
fd8888f6f037 Naming style changes, finally got tired of most of the typedefs. Also the
Timo Sirainen <tss@iki.fi>
parents: 764
diff changeset
199 uoff_t start_offset, uoff_t v_size,
3863
55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool.
Timo Sirainen <tss@iki.fi>
parents: 3647
diff changeset
200 bool autoclose_fd)
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
201 {
903
fd8888f6f037 Naming style changes, finally got tired of most of the typedefs. Also the
Timo Sirainen <tss@iki.fi>
parents: 764
diff changeset
202 struct mmap_istream *mstream;
1197
e86f259048cf o_stream_send_istream() is now safe to use for moving data within file.
Timo Sirainen <tss@iki.fi>
parents: 1146
diff changeset
203 struct istream *istream;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
204 struct stat st;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
205
2439
7d6ca38c8c37 Added mmap_get_page_size() call which basically returns getpagesize() but
Timo Sirainen <tss@iki.fi>
parents: 2431
diff changeset
206 if (mmap_pagemask == 0)
7d6ca38c8c37 Added mmap_get_page_size() call which basically returns getpagesize() but
Timo Sirainen <tss@iki.fi>
parents: 2431
diff changeset
207 mmap_pagemask = mmap_get_page_size()-1;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
208
21365
cc6e4b239003 istream-mmap: Mark stream eof on error and copy errno
Aki Tuomi <aki.tuomi@dovecot.fi>
parents: 19552
diff changeset
209 mstream = i_new(struct mmap_istream, 1);
cc6e4b239003 istream-mmap: Mark stream eof on error and copy errno
Aki Tuomi <aki.tuomi@dovecot.fi>
parents: 19552
diff changeset
210
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
211 if (v_size == 0) {
21365
cc6e4b239003 istream-mmap: Mark stream eof on error and copy errno
Aki Tuomi <aki.tuomi@dovecot.fi>
parents: 19552
diff changeset
212 if (fstat(fd, &st) < 0) {
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
213 i_error("i_stream_create_mmap(): fstat() failed: %m");
21365
cc6e4b239003 istream-mmap: Mark stream eof on error and copy errno
Aki Tuomi <aki.tuomi@dovecot.fi>
parents: 19552
diff changeset
214 mstream->istream.istream.eof = TRUE;
cc6e4b239003 istream-mmap: Mark stream eof on error and copy errno
Aki Tuomi <aki.tuomi@dovecot.fi>
parents: 19552
diff changeset
215 mstream->istream.istream.stream_errno = errno;
cc6e4b239003 istream-mmap: Mark stream eof on error and copy errno
Aki Tuomi <aki.tuomi@dovecot.fi>
parents: 19552
diff changeset
216 } else {
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
217 v_size = st.st_size;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
218 if (start_offset > v_size)
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
219 start_offset = v_size;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
220 v_size -= start_offset;
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
221 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
222 }
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
223
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
224 mstream->autoclose_fd = autoclose_fd;
1870
c972ea085643 istream rewrite. instead of directly setting any limits to stream, you now
Timo Sirainen <tss@iki.fi>
parents: 1861
diff changeset
225 mstream->v_size = v_size;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
226
6420
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
227 mstream->istream.iostream.close = i_stream_mmap_close;
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
228 mstream->istream.iostream.destroy = i_stream_mmap_destroy;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
229
5479
b3999af65adb If istream implementation doesn't set set_max_buffer_size() or stat(),
Timo Sirainen <tss@iki.fi>
parents: 5106
diff changeset
230 mstream->istream.max_buffer_size = block_size;
6420
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
231 mstream->istream.read = i_stream_mmap_read;
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
232 mstream->istream.seek = i_stream_mmap_seek;
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
233 mstream->istream.sync = i_stream_mmap_sync;
a8b515e1a26f Removed _ prefixes from function names.
Timo Sirainen <tss@iki.fi>
parents: 6415
diff changeset
234 mstream->istream.stat = i_stream_mmap_stat;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
235
9540
622509c562e8 Added struct istream.readable_fd, which is used to determine if sendfile() can be used.
Timo Sirainen <tss@iki.fi>
parents: 8590
diff changeset
236 mstream->istream.istream.readable_fd = TRUE;
7032
85cf52f0bc64 i_stream_create() now takes parent stream as parameter and internally sets
Timo Sirainen <tss@iki.fi>
parents: 6429
diff changeset
237 mstream->istream.abs_start_offset = start_offset;
85cf52f0bc64 i_stream_create() now takes parent stream as parameter and internally sets
Timo Sirainen <tss@iki.fi>
parents: 6429
diff changeset
238 istream = i_stream_create(&mstream->istream, NULL, fd);
1197
e86f259048cf o_stream_send_istream() is now safe to use for moving data within file.
Timo Sirainen <tss@iki.fi>
parents: 1146
diff changeset
239 istream->mmaped = TRUE;
5106
81394e71f92a Added istream->blocking setting. It's now used to assert-crash early if a
Timo Sirainen <tss@iki.fi>
parents: 4766
diff changeset
240 istream->blocking = TRUE;
3239
904a268921af Added seekable variable to struct istream.
Timo Sirainen <tss@iki.fi>
parents: 2439
diff changeset
241 istream->seekable = TRUE;
1197
e86f259048cf o_stream_send_istream() is now safe to use for moving data within file.
Timo Sirainen <tss@iki.fi>
parents: 1146
diff changeset
242 return istream;
764
f57c52738f90 Renamed IBuffer and OBuffer to IStream and OStream which describes their
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
243 }