Increasing file I/O performance with mmap - Part 2

This the continuation of my previous blog http://sivan11in.blogspot.in/2013/10/increasing-file-io-performance-with-mmap.html .

mmap as I said earlier, map the entire file into memory as big long array.
mmap's operation is limited by the following system variables.

1. The amount of free memory the system has at that of point of time.
2. The amount of fragmentation in the file which you are aiming at.
3. The reading mode of the file i.e sequential / random.

There are lot of flags available with mmap which can help you to specify what level of mapping you want mmap to do while mapping you file.

For example, if you mapping a file for a sequential read operation, then you can use MAP_POPULATE as the flag for mmap. This will make the mmap to perform some thing knows as readhead which will read the file for the specified length and populate the pagetable so that your read operations does not have to go through a traditional I/O call . This will improve the performance to a greater extent.

#include "stdio.h"
#include "stdlib.h"
#include "sys types.h"
#include "sys stat.h"
#include "unistd.h"
#include "fcntl.h"
#include "sys/mman.h" // This is the header file for mmap

int main(int argc, char *argv[])
{

char *pMapped;
struct stat fileStat;

if (argc != 3 || (fdin = open (argv[1], O_RDONLY) < 0)
{
   printf ("usage:  ");
   exit(-1);
}

 fstat (fdin,&fileStat)

 printf("\n Mapping file %s for size %d", argv[1], fileStat.st_size);

    /* Now the file is ready to be mmapped.
     */
    pMapped = mmap(0, fileStat.st_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_POPULATE, fdin, 0);
    if (pMapped == MAP_FAILED) {
 close(fdin);
 perror("Error mmapping the file");
 exit(EXIT_FAILURE);
    }
    
 /* Here afterwards, pMapped can be used as a normal array pointer */
printf("accessing 5 element of the file %c", pMapped[4]);
    
    if (munmap(pMapped, fileStat.st_size) == -1) {
 perror("Error un-mmapping the file");
 /* Decide here whether to close(fdin) and exit() or not. Depends... */
    }

    /* Un-mmaping doesn't close the file, so we still need to do that.
     */
    close(fdin);
    return 0;
}

Some explanations...

PROT_READ | PROT_WRITE means mmapped file can be either read / written.
MAP_PRIVATE | MAP_POPULATE means the mapped portion is private only to this process. It is not shared with other processes. If you want to do so, replace MAP_PRIVATE with MAP_SHARED.
MAP_POPULATE as explained earlier performs readahead.

It is always preferable to use stat structure to get the file size rather than lseek from 0 to EOF.

that's the end of tutorial :-)

Increasing file I/O performance with mmap

While I decided to start blogging "again", the first point which came to my mind is to blog about mmap.
mmap is the recent magic word which I learnt and I deeply wanted to tell to this all who see my blog ( If any :-) ).

mmap is a system call provided by linux kernel. In simple english, mmap actually _pretends_ the file system as a normal memory to the application. That said, application can start accessing the file as a really really long array. When you execute mmap with the set of argument [ I dont wanna talk about the semantics of mmap.. you have google and wiki gurus... they have much tell you about the sementics ] , mmap just returns a char* . From now on, all your activities you wanna do with file, just do with the char* arithmetic.

I hear your questions about, lseek . That is so simple. The pointer returned always be at the first offset from the start position which you have given in mmap declaration. [ If you have given 0 in the mmap declaration, the char * is equal to lseek with 0 ]. So all you have to do is char * + length of the bytes which you need to point at.

Remember, when you do mmap, the kernel does not actually read the file. Only when you "ask" the kernel to read, i.e when you "use" that pointer, the paging happens. i.e OS actually reads the disk and populates the contents of the file. This whole process happens just like that so that you wont feel any difference.

So what are the advantages of using  mmap ?

1. If you are dealing with large fat files i.e files with GBs, typical file I/O read / write does not help you. You need to have compiled your application with LFS support [ Large File System ] support. Else your lseek / fopen etc will fail.
2. For 64bit application, all your 32bit file handling wont help.
3. You have a really big file and you are sporadically searching something in file at random locations .

For all the above cases (atleast) you need to depend on mmap.

Next post, I will talk about tips and tricks of using mmap...