NFS Performance measurement for Linux 2.2 client
21 Apr 1999 T. Ichihara (RIKEN) updated 13 Oct 1999 T. Ichihara (RIKEN)Preliminary
NFS Performance test [Draft] (Linux client - Linux Server v.s. Linux Client - Solaris2.6 Server) Purpose Compare the performance of NFS server (SUN Solaris v.s. i386 Linux) for Linux NFS client, because we felt that the NFS-write performance for Solaris 2.6 NFS server from Linux NFS client is very slow to our experience so far. ----------------------------------------------------------------------- Test method 1.Fortran unformatted read & Write for 1MB-size variables [Sample test program] fwrite_10MB.f, fread_10MB.f Test method 2. bonnie-1.0-4cl.i386.html benchmark Measured at: RIKEN Wako CC-J phase 0.2 system ----------------------------------------------------------------------(1) Linux 2.2.4 NFS client + Solaris 2.6 NFS Server
NFS Client: Linux 2.2.4 Dual Pentium II 450MHz, 256 MB Memory NFS Server 1: SUN E450: Solaris2.6 (Dual Ultra2 Sparc 400MHz) 1280 MB Memory 18GB Bare Disk (Seagate UW SCSI), Load average : less than 0.5 (Netperf: TCP STREAM 66.5 Mbps from NFS client to server) Test 1. 10 MB NFS write 15.94 s 0.62 MB/s <- ! slow 100 MB NFS write 206.94 s 0.48 MB/s <- ! slow 200 MB NFS write 377.88 s 0.53 MB/s <- ! slow 400 MB NFS write 643.04 s 0.62 MB/s <- ! slow 1000 MB NFS write 1971.22 s 0.51 MB/s <- ! slow 100 MB NFS read 15.52 s 6.5 MB/s 200 MB NFS read 41.00 s 4.9 MB/s 400 MB NFS read 82.34 s 4.9 MB/s 1000 MB NFS read 199.95 s 5.0 MB/s Test 2. /usr/sbin/bonnie (default parameter: 100 MB file size) results -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 477 5.7 608 1.2 794 1.7 9688 99.9 235567 101.2 481.4 4.6 /usr/sbin/bonnie -s 1024 ( 1000 MB file size) results -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 1024 787 9.7 508 1.1 670 2.4 3585 43.8 5594 10.7 182.8 6.6(2) Linux 2.2.4 NFS client + Linux 2.2.4 NFS server
NFS Client: Linux 2.2.4 Dual Pentium II 450 MHz, 256 MB Memory NFS Server 2: Linux 2.2.4 Dual Pentium II 450 MHz, 256 MB Memory 18GB Bare Disk (Seagate UW SCSI), Load average : less than 0.5 (Netperf: TCP STREAM 66.8 Mbps from NFS client to server) Test. 1 10 MB NFS write 1.37 s 7.3 MB/s 100 MB NFS write 13.68 s 7.3 MB/s 200 MB NFS write 27.54 s 7.3 MB/s 300 MB NFS write 55.83 s 5.4 MB/s 400 MB NFS write 78.60 s 5.1 MB/s 1000 MB NFS write 229.69 s 4.3 MB/s 2000 MB NFS write 460.17 s 4.3 MB/s 100 MB NFS read 25.28 s 4.0 MB/s 200 MB NFS read 49.27 s 4.1 MB/s 300 MB NFS read 73.80 s 4.1 MB/s 400 MB NFS read 108.94 s 4.2 MB/s 1000 MB NFS read 229.26 s 4.4 MB/s 2000 MB NFS read 533.24 s 3.8 MB/s Test. 2 /usr/sbin/bonnie (default parameter: 100 MB file size) results -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 4001 50.5 7464 14.2 7133 15.3 9697 99.9 232212 99.8 9179.7 98.7 /usr/sbin/bonnie -s 1024 ( 1000 MB file size) results -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 1024 3306 42.2 5049 10.1 2130 7.0 3723 45.5 5112 9.4 55.9 1.4 (Linux Software environment) Basic Linux Dis. : Redhat 5.2 Linux kernel : 2.2.4 /usr/sbin/rpc.nfsd -v : Universal NFS Server 2.2beta37 /usr/sbin/rpc.mountd -v : Universal NFS Server 2.2beta37(3) Cf.
NFS client: rlnx01.rhic NFS server: radsm.rhic (/phenix/workareas/ file system) measured date : Tue Apr 20 00:07:28 EDT 1999 Test 1. (load avarage =3.27 2-staf running) 10 MB NFS write 88.17 s 0.11 MB/s 100 MB NFS write 1037.17 s 0.10 MB/s 10 MB NFS read 11.05 s 0.90 MB/s 100 MB NFS read 121.65 s 0.82 MB/s Test 2. (load avaragee = 0.0 nojobs running) /usr/sbin/bonnie (default parameter: 100 MB file size) results -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 138 2.7 142 0.6 111 1.1 806 13.6 200039 99.6 56.8 3.7Result Summary
NFS Client : Linux (Linux 2.2.4) NFS Write NFS read Netperf ------------------+---------------------------------------------- SUN NFS server | 0.5 MB/s 5-6.5 MB/s 66.5 Mbps Linux NFS Server | 4.3-7.3 MB/s 4 MB/s 66.8 Mbps For the Linux NFS client (Linux 2.2.4), the NFS-write performance of the Linux NFS server is about 10 times higher than that of SUN Solaris 2.6 NFS server in this measurement. Further NFS performance measurements are in plan. NB: This does not mean that the NFS server performance of Linux is generally better than that of Solaris 2.6. (Sun Solaris 2.6 supports NFS version 3 while Linux supports only NFS version 2.0.) This is a simple comparison of the performance of the NFS server of SUN Solaris 2.6 and Linux 2.2.4 in the condition that the NFS client is Linux 2.2.4.
NFSv3 Kernel Patch for Linux
NFSv3 Kernel patch for Linux is available and this significantly improves the Linux NFS client performance. This is our current solution.
Linux NFSv3 kernel patch is available at the URL of http://www.fys.uio.no/~trondmy/src/ (primary site) or ftp://ftp.riken.go.jp/pub/Linux/nfsv3 (ftp mirror) N.B. 0 apply NFSV3 patch (patch -p1 nfsv3_patch at /usr/src/linux ) N.B. 1 CONFIG_NFS_V3 should be defined when compiling the kernel to activate the NFSv3 code. (select NFS V3 at "Network File System" Menu at the main configure ) N.B. 2 /bin/mount should also be replaced to use NFSv3 client function. N.B. 3 "more /proc/mounts" will show the NFS mount status. When the NFSv3 function works, "v3" status can be seen like as follows: % more /proc/mounts host:/ccj/u /ccj/u nfs rw,v3,rsize=8192,wsize=8192,intr,addr=ccjsun 0 0 - this shows that the file system /ccj/u is mounted by NFSv3 host:/ccj/u /ccj/u nfs rw,rsize=8192,wsize=8192,intr,addr=ccjsun 0 0 - this shows that the file system /ccj/u is mounted by standard NFSv2 Linux NFSv3 Performance example =============================== NFS Server : SUN Solaris 2.6 (SUN E450 2 CPU) NFS Client : Linux (Dual PII 450 MHz) Linux kernel 2.2.10 with NFSv3 Patch -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 6559 89.2 6554 16.3 6791 19.1 9650 100.0 252616 101.1 1262.3 1 ^^^^ Linux kernel 2.2.10 without NFSv3 Patch -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 374 4.6 483 1.3 523 1.5 9712 100.0 254025 99.2 571.5 6.1 ^^^ NFS Server: SUN E450 (Solaris 2.6) NFS Write NFS read --------------------------------+------------------------------- linux Kernel 2.2.10 | 0.4 MB/s 9.7 MB/s linux Kernel 2.2.10 with NFSv3 | 6.6 MB/s 9.7 MB/s