Results for MPI perforance tests on sp2
Contents
memcpy
Determining delivered memory performance
mpicc -o memcpy -O memcpy.c
Size (bytes) Time (sec) Rate (MB/sec)
4 0.000000 18.985126
8 0.000000 37.941395
16 0.000000 76.032619
32 0.000000 70.972320
64 0.000001 109.090656
128 0.000001 149.132699
256 0.000001 182.660843
512 0.000002 205.801349
1024 0.000005 219.709235
2048 0.000009 227.390348
4096 0.000018 231.436011
8192 0.000035 233.497731
16384 0.000070 234.552805
32768 0.000139 235.082143
65536 0.000278 235.349463
131072 0.000574 228.418066
262144 0.001592 164.619353
524288 0.003165 165.676686
1048576 0.006346 165.228959
2097152 0.012861 163.057514
Running job under PBS
Determining delivered memory performance with unaligned data
mpicc -o memcpy memcpy.c
Size (bytes) Time (sec) Rate (MB/sec)
4 0.000000 15.642843
8 0.000000 29.534463
16 0.000000 56.050671
32 0.000001 56.050669
64 0.000001 86.871068
128 0.000001 119.808007
256 0.000002 147.840661
512 0.000003 167.424287
1024 0.000006 179.298258
2048 0.000011 185.889342
4096 0.000022 189.365272
8192 0.000043 191.148339
16384 0.000085 192.051572
32768 0.000170 192.512289
65536 0.000340 192.745864
131072 0.000698 187.880932
262144 0.001853 141.450967
524288 0.003683 142.344819
1048576 0.007368 142.308116
2097152 0.014944 140.334516
Running job under PBS
pingpong
Benchmarking point to point performance
mpicc -o pingpong -O pingpong.c
Kind n time (sec) Rate (MB/sec)
Send/Recv 1 0.000053 0.150089
Send/Recv 2 0.000054 0.298215
Send/Recv 4 0.000054 0.596500
Send/Recv 8 0.000054 1.184371
Send/Recv 16 0.000055 2.310024
Send/Recv 32 0.000071 3.630687
Send/Recv 64 0.000084 6.099959
Send/Recv 128 0.000101 10.126798
Send/Recv 256 0.000141 14.563124
Send/Recv 512 0.000213 19.230048
Send/Recv 1024 0.000423 19.361279
Send/Recv 2048 0.000674 24.323040
Send/Recv 4096 0.001171 27.989792
Send/Recv 8192 0.002097 31.246490
Send/Recv 16384 0.004030 32.524977
Send/Recv 32768 0.007784 33.679180
Send/Recv 65536 0.015430 33.977823
Send/Recv 131072 0.030680 34.178072
Send/Recv 262144 0.061428 34.140008
Send/Recv 524288 0.122645 34.198834
Send/Recv 1048576 0.244523 34.305987
Running job under PBS
Benchmarking point to point performance with nonblocking operations
mpicc -o pingpong -O pingpong.c
Kind n time (sec) Rate (MB/sec)
Isend/Irecv 1 0.000063 0.125993
Isend/Irecv 2 0.000063 0.252664
Isend/Irecv 4 0.000063 0.504331
Isend/Irecv 8 0.000064 1.000644
Isend/Irecv 16 0.000065 1.968846
Isend/Irecv 32 0.000081 3.141229
Isend/Irecv 64 0.000092 5.575469
Isend/Irecv 128 0.000112 9.133245
Isend/Irecv 256 0.000153 13.353618
Isend/Irecv 512 0.000225 18.239913
Isend/Irecv 1024 0.000420 19.483886
Isend/Irecv 2048 0.000685 23.914758
Isend/Irecv 4096 0.001165 28.116783
Isend/Irecv 8192 0.002098 31.235881
Isend/Irecv 16384 0.004051 32.358663
Isend/Irecv 32768 0.007869 33.313720
Isend/Irecv 65536 0.015441 33.954058
Isend/Irecv 131072 0.030745 34.105911
Isend/Irecv 262144 0.061168 34.284962
Isend/Irecv 524288 0.122713 34.179858
Isend/Irecv 1048576 0.244238 34.346092
Running job under PBS
Benchmarking point to point performance with nonblocking operations, head-to-head
mpicc -o pingpong -O pingpong.c
Kind n time (sec) Rate (MB/sec)
head-to-head Isend/Irecv 1 0.000078 0.204488
head-to-head Isend/Irecv 2 0.000078 0.407942
head-to-head Isend/Irecv 4 0.000079 0.810322
head-to-head Isend/Irecv 8 0.000079 1.615431
head-to-head Isend/Irecv 16 0.000081 3.148363
head-to-head Isend/Irecv 32 0.000097 5.289917
head-to-head Isend/Irecv 64 0.000111 9.196503
head-to-head Isend/Irecv 128 0.000132 15.482894
head-to-head Isend/Irecv 256 0.000171 23.922904
head-to-head Isend/Irecv 512 0.000285 28.756468
head-to-head Isend/Irecv 1024 0.000560 29.238868
head-to-head Isend/Irecv 2048 0.000963 34.022586
head-to-head Isend/Irecv 4096 0.001630 40.214152
head-to-head Isend/Irecv 8192 0.003025 43.322428
head-to-head Isend/Irecv 16384 0.005868 44.672340
head-to-head Isend/Irecv 32768 0.011529 45.477457
head-to-head Isend/Irecv 65536 0.022791 46.008286
head-to-head Isend/Irecv 131072 0.045532 46.059315
head-to-head Isend/Irecv 262144 0.090737 46.224759
head-to-head Isend/Irecv 524288 0.181549 46.205866
head-to-head Isend/Irecv 1048576 0.363547 46.148720
Running job under PBS
Benchmarking point to point performance with unaligned data
mpicc -o pingpong -O pingpong.c
Kind char n time (sec) Rate (MB/sec)
Send/Recv 1 0.000053 0.018697
Send/Recv 2 0.000053 0.037477
Send/Recv 4 0.000053 0.075319
Send/Recv 8 0.000053 0.149838
Send/Recv 16 0.000053 0.302274
Send/Recv 32 0.000054 0.593275
Send/Recv 64 0.000054 1.179579
Send/Recv 128 0.000057 2.264127
Send/Recv 256 0.000071 3.590043
Send/Recv 512 0.000087 5.891832
Send/Recv 1024 0.000107 9.547780
Send/Recv 2048 0.000141 14.499118
Send/Recv 4096 0.000210 19.485052
Send/Recv 8192 0.000410 19.967704
Send/Recv 16384 0.000672 24.398197
Send/Recv 32768 0.001156 28.357368
Send/Recv 65536 0.002106 31.121664
Send/Recv 131072 0.004029 32.535675
Send/Recv 262144 0.007798 33.618711
Send/Recv 524288 0.015488 33.851813
Send/Recv 1048576 0.030682 34.175107
Kind double n time (sec) Rate (MB/sec)
Send/Recv 1 0.000053 0.149679
Send/Recv 2 0.000053 0.299734
Send/Recv 4 0.000054 0.593250
Send/Recv 8 0.000054 1.182612
Send/Recv 16 0.000056 2.304716
Send/Recv 32 0.000071 3.623020
Send/Recv 64 0.000086 5.967946
Send/Recv 128 0.000103 9.981203
Send/Recv 256 0.000142 14.424649
Send/Recv 512 0.000207 19.823354
Send/Recv 1024 0.000407 20.106769
Send/Recv 2048 0.000684 23.938342
Send/Recv 4096 0.001158 28.290344
Send/Recv 8192 0.002119 30.932905
Send/Recv 16384 0.004069 32.210457
Send/Recv 32768 0.007951 32.971029
Send/Recv 65536 0.015507 33.809382
Send/Recv 131072 0.030882 33.953783
Send/Recv 262144 0.061395 34.158449
Send/Recv 524288 0.122433 34.257915
Send/Recv 1048576 0.245376 34.186701
Kind int n time (sec) Rate (MB/sec)
Send/Recv 1 0.000053 0.074976
Send/Recv 2 0.000053 0.149942
Send/Recv 4 0.000054 0.298942
Send/Recv 8 0.000054 0.596341
Send/Recv 16 0.000054 1.178917
Send/Recv 32 0.000055 2.321520
Send/Recv 64 0.000071 3.630004
Send/Recv 128 0.000084 6.094201
Send/Recv 256 0.000102 10.004071
Send/Recv 512 0.000145 14.141208
Send/Recv 1024 0.000211 19.401973
Send/Recv 2048 0.000419 19.560066
Send/Recv 4096 0.000669 24.486165
Send/Recv 8192 0.001159 28.282715
Send/Recv 16384 0.002124 30.860439
Send/Recv 32768 0.004062 32.269040
Send/Recv 65536 0.007892 33.216738
Send/Recv 131072 0.015544 33.729800
Send/Recv 262144 0.030750 34.100282
Send/Recv 524288 0.061239 34.245436
Send/Recv 1048576 0.122615 34.206981
Running job under PBS
Benchmarking point to point performance with contention
mpicc -o pingpong -O pingpong.c
Kind (np=2) n time (sec) Rate (MB/sec)
Send/Recv 1 0.000054 0.149366
Send/Recv 2 0.000054 0.296665
Send/Recv 4 0.000054 0.594678
Send/Recv 8 0.000054 1.181097
Send/Recv 16 0.000056 2.290092
Send/Recv 32 0.000071 3.584928
Send/Recv 64 0.000085 6.027016
Send/Recv 128 0.000100 10.224116
Send/Recv 256 0.000142 14.442878
Send/Recv 512 0.000210 19.512888
Send/Recv 1024 0.000422 19.406569
Send/Recv 2048 0.000680 24.089690
Send/Recv 4096 0.001163 28.170564
Send/Recv 8192 0.002098 31.243885
Send/Recv 16384 0.004012 32.667039
Send/Recv 32768 0.007813 33.554218
Send/Recv 65536 0.015561 33.692003
Send/Recv 131072 0.030591 34.277481
Send/Recv 262144 0.061389 34.161628
Send/Recv 524288 0.122552 34.224667
Send/Recv 1048576 0.245654 34.148081
Running job under PBS
Kind (np=4) n time (sec) Rate (MB/sec)
Send/Recv 1 0.000054 0.148377
Send/Recv 2 0.000054 0.298231
Send/Recv 4 0.000054 0.589707
Send/Recv 8 0.000054 1.175876
Send/Recv 16 0.000056 2.290547
Send/Recv 32 0.000071 3.618911
Send/Recv 64 0.000086 5.969627
Send/Recv 128 0.000101 10.144532
Send/Recv 256 0.000141 14.510244
Send/Recv 512 0.000212 19.331010
Send/Recv 1024 0.000416 19.675755
Send/Recv 2048 0.000676 24.254625
Send/Recv 4096 0.001160 28.248276
Send/Recv 8192 0.002121 30.902458
Send/Recv 16384 0.004045 32.399757
Send/Recv 32768 0.007877 33.278618
Send/Recv 65536 0.015517 33.788655
Send/Recv 131072 0.030811 34.032479
Send/Recv 262144 0.061475 34.113629
Send/Recv 524288 0.122883 34.132372
Send/Recv 1048576 0.245223 34.208087
Running job under PBS
Kind (np=8) n time (sec) Rate (MB/sec)
Send/Recv 1 0.000054 0.148451
Send/Recv 2 0.000054 0.296799
Send/Recv 4 0.000054 0.587943
Send/Recv 8 0.000055 1.173255
Send/Recv 16 0.000056 2.274496
Send/Recv 32 0.000072 3.570874
Send/Recv 64 0.000086 5.971542
Send/Recv 128 0.000103 9.950717
Send/Recv 256 0.000142 14.386233
Send/Recv 512 0.000213 19.270756
Send/Recv 1024 0.000420 19.500122
Send/Recv 2048 0.000679 24.119833
Send/Recv 4096 0.001164 28.144551
Send/Recv 8192 0.002122 30.886619
Send/Recv 16384 0.004038 32.457222
Send/Recv 32768 0.007872 33.300231
Send/Recv 65536 0.015558 33.698473
Send/Recv 131072 0.030774 34.073882
Send/Recv 262144 0.061564 34.064832
Send/Recv 524288 0.122607 34.209301
Send/Recv 1048576 0.246236 34.067417
Running job under PBS
Command to clear nodes took too long.
Please contact system administrator.
Kind (np=16) n time (sec) Rate (MB/sec)
Send/Recv 1 0.000054 0.147868
Send/Recv 2 0.000054 0.295488
Send/Recv 4 0.000055 0.586556
Send/Recv 8 0.000055 1.166706
Send/Recv 16 0.000056 2.271948
Send/Recv 32 0.000072 3.568466
Send/Recv 64 0.000087 5.918733
Send/Recv 128 0.000105 9.778657
Send/Recv 256 0.000146 14.056682
Send/Recv 512 0.000225 18.208489
Send/Recv 1024 0.000442 18.554403
Send/Recv 2048 0.000718 22.826492
Send/Recv 4096 0.001253 26.160770
Send/Recv 8192 0.002282 28.715994
Send/Recv 16384 0.004405 29.758402
Send/Recv 32768 0.008663 30.260100
Send/Recv 65536 0.017099 30.661686
Send/Recv 131072 0.034008 30.833182
Send/Recv 262144 0.068180 30.759016
Send/Recv 524288 0.135852 30.874015
Send/Recv 1048576 0.270954 30.959542
Running job under PBS
Kind (np=32) n time (sec) Rate (MB/sec)
Send/Recv 1 0.000055 0.146436
Send/Recv 2 0.000055 0.293164
Send/Recv 4 0.000055 0.580721
Send/Recv 8 0.000055 1.153623
Send/Recv 16 0.000058 2.202097
Send/Recv 32 0.000074 3.438773
Send/Recv 64 0.000088 5.805757
Send/Recv 128 0.000108 9.478347
Send/Recv 256 0.000156 13.103704
Send/Recv 512 0.000334 12.263935
Send/Recv 1024 0.000503 16.283452
Send/Recv 2048 0.000829 19.762379
Send/Recv 4096 0.001534 21.359406
Send/Recv 8192 0.002848 23.013458
Send/Recv 16384 0.005491 23.869518
Send/Recv 32768 0.010778 24.321602
Send/Recv 65536 0.021273 24.645701
Send/Recv 131072 0.042271 24.806096
Send/Recv 262144 0.083974 24.973866
Send/Recv 524288 0.168072 24.955330
Send/Recv 1048576 0.336926 24.897471
Running job under PBS
Kind (np=64) n time (sec) Rate (MB/sec)
Send/Recv 1 0.000055 0.146161
Send/Recv 2 0.000054 0.294086
Send/Recv 4 0.000055 0.585374
Send/Recv 8 0.000055 1.156581
Send/Recv 16 0.000056 2.300315
Send/Recv 32 0.000071 3.595709
Send/Recv 64 0.000086 5.974503
Send/Recv 128 0.000108 9.522260
Send/Recv 256 0.000228 8.984426
Send/Recv 512 0.000314 13.050301
Send/Recv 1024 0.000488 16.776142
Send/Recv 2048 0.000796 20.594557
Send/Recv 4096 0.001487 22.030759
Send/Recv 8192 0.002909 22.531996
Send/Recv 16384 0.005741 22.829176
Send/Recv 32768 0.011942 21.951156
Send/Recv 65536 0.024091 21.763245
Send/Recv 131072 0.047464 22.092173
Send/Recv 262144 0.094512 22.189308
Send/Recv 524288 0.182551 22.976102
Send/Recv 1048576 0.370842 22.620428
barrier
Benchmarking collective barrier
mpicc -o barrier -O barrier.c
Kind np time (sec)
Barrier 1 0.000009
Running job under PBS
Barrier 2 0.000083
Running job under PBS
Barrier 4 0.000148
Running job under PBS
Barrier 8 0.000217
Running job under PBS
Barrier 16 0.000297
Running job under PBS
Barrier 32 0.000391
Running job under PBS
Barrier 64 0.000564
Benchmarking collective Allreduce
mpicc -o barrier -O barrier.c
Kind np time (sec)
Allreduce 1 0.000019
Running job under PBS
Allreduce 2 0.000107
Running job under PBS
Allreduce 4 0.000182
Running job under PBS
Allreduce 8 0.000265
Running job under PBS
Allreduce 16 0.000362
Running job under PBS
Allreduce 32 0.000476
Running job under PBS
Allreduce 64 0.000667
vector
Comparing the performance of MPI vector datatypes
mpicc -o vector -O vector.c
Kind n stride time (sec) Rate (MB/sec)
Vector 1000 24 0.002031 3.939534
Struct 1000 24 0.002665 3.002139
User 1000 24 0.000536 14.933010
User(add) 1000 24 0.000534 14.980330
Running job under PBS
circulate
Pipelining pitfalls
mpicc -c -O circulate.c
mpicc -o circulate -O circulate.o -lm
For n = 20000, m = 20000, T_comm = 0.012568, T_compute = 0.025471, sum = 0.038039, T_both = 0.031753
Running job under PBS
For n = 500, m = 500, T_comm = 0.000149, T_compute = 0.000630, sum = 0.000779, T_both = 0.000771
Running job under PBS
3way
Exploring the cost of synchronization delays
mpicc -c -O bad.c
mpicc -o bad -O bad.o -lm
[2] Litsize = 8, Time for first send = 0.000058, for second = 0.000030
Running job under PBS
[2] Litsize = 9, Time for first send = 0.000051, for second = 0.000030
Running job under PBS
[2] Litsize = 511, Time for first send = 0.000173, for second = 0.000138
Running job under PBS
[2] Litsize = 512, Time for first send = 0.000172, for second = 0.000141
Running job under PBS
[2] Litsize = 513, Time for first send = 0.000341, for second = 0.001797
Running job under PBS
jacobi
Jacobi Iteration - Example Parallel Mesh
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
send/recv: 25 iterations in 0.014018 secs (1.997457 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
send/recv: 25 iterations in 0.858319 secs (26.723861 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
send/recv: 25 iterations in 0.017718 secs (3.160579 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
send/recv: 25 iterations in 1.814446 secs (25.283312 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
send/recv: 25 iterations in 0.022098 secs (5.068343 MFlops); diffnorm 0.080560, m=7 n=130 np=64
send/recv: 25 iterations in 3.511754 secs (26.126662 MFlops); diffnorm 0.474303, m=4098 n=130 np=64
Jacobi Iteration - Shift up and down
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
shift/sendrecv: 25 iterations in 0.014787 secs (1.893517 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
shift/sendrecv: 25 iterations in 0.144997 secs (158.193181 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
shift/sendrecv: 25 iterations in 0.019105 secs (2.931166 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
shift/sendrecv: 25 iterations in 0.175082 secs (262.021187 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
shift/sendrecv: 25 iterations in 0.024481 secs (4.575037 MFlops); diffnorm 0.080560, m=7 n=130 np=64
shift/sendrecv: 25 iterations in 0.168200 secs (545.483704 MFlops); diffnorm 0.474303, m=4098 n=130 np=64
Jacobi Iteration - Exchange head-to-head
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
head-to-head sendrecv: 25 iterations in 0.014324 secs (1.954696 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
head-to-head sendrecv: 25 iterations in 0.153932 secs (149.011736 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
head-to-head sendrecv: 25 iterations in 0.018076 secs (3.098103 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
head-to-head sendrecv: 25 iterations in 0.181382 secs (252.920530 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
head-to-head sendrecv: 25 iterations in 0.023036 secs (4.861855 MFlops); diffnorm 0.080560, m=7 n=130 np=64
head-to-head sendrecv: 25 iterations in 0.172437 secs (532.080162 MFlops); diffnorm 0.474303, m=4098 n=130 np=64
Jacobi Iteration - Nonblocking send/recv
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
irecv/isend: 25 iterations in 0.013462 secs (2.079867 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
irecv/isend: 25 iterations in 0.147150 secs (155.878532 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
irecv/isend: 25 iterations in 0.020551 secs (2.724965 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
irecv/isend: 25 iterations in 0.178002 secs (257.722404 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
irecv/isend: 25 iterations in 0.023265 secs (4.814161 MFlops); diffnorm 0.080560, m=7 n=130 np=64
irecv/isend: 25 iterations in 0.175803 secs (521.894369 MFlops); diffnorm 0.474303, m=4098 n=130 np=64
Jacobi Iteration - Nonblocking send/recv for receiver pull
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
isend/irecv: 25 iterations in 0.014331 secs (1.953823 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
isend/irecv: 25 iterations in 0.139625 secs (164.280095 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
isend/irecv: 25 iterations in 0.019518 secs (2.869161 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
isend/irecv: 25 iterations in 0.168687 secs (271.954566 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
isend/irecv: 25 iterations in 0.024794 secs (4.517204 MFlops); diffnorm 0.080560, m=7 n=130 np=64
isend/irecv: 25 iterations in 0.155092 secs (591.586929 MFlops); diffnorm 0.474303, m=4098 n=130 np=64
Jacobi Iteration - Synchronous send
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
ssend/irecv: 25 iterations in 0.019663 secs (1.423976 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
ssend/irecv: 25 iterations in 0.258036 secs (88.893005 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
ssend/irecv: 25 iterations in 0.023209 secs (2.412857 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
ssend/irecv: 25 iterations in 0.175369 secs (261.593047 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
ssend/irecv: 25 iterations in 0.031128 secs (3.598090 MFlops); diffnorm 0.080560, m=7 n=130 np=64
b1313.nas.nasa.gov b1311.nas.nasa.gov b1309.nas.nasa.gov b1307.nas.nasa.gov b1305.nas.nasa.gov b1303.nas.nasa.gov b1215.nas.nasa.gov b1213.nas.nasa.gov b1211.nas.nasa.gov b1209.nas.nasa.gov b1207.nas.nasa.gov b1205.nas.nasa.gov b1203.nas.nasa.gov b1115.nas.nasa.gov b1113.nas.nasa.gov b1111.nas.nasa.gov b1109.nas.nasa.gov b1107.nas.nasa.gov b1105.nas.nasa.gov b1103.nas.nasa.gov b1015.nas.nasa.gov b1013.nas.nasa.gov b1011.nas.nasa.gov b1009.nas.nasa.gov b1007.nas.nasa.gov b1005.nas.nasa.gov b1003.nas.nasa.gov b1001.nas.nasa.gov b0915.nas.nasa.gov b0913.nas.nasa.gov b0911.nas.nasa.gov b0909.nas.nasa.gov b0907.nas.nasa.gov b0905.nas.nasa.gov b0903.nas.nasa.gov b0815.nas.nasa.gov b0813.nas.nasa.gov b0811.nas.nasa.gov b0809.nas.nasa.gov b0807.nas.nasa.gov b0805.nas.nasa.gov b0803.nas.nasa.gov b0801.nas.nasa.gov b0715.nas.nasa.gov b0713.nas.nasa.gov b0711.nas.nasa.gov b0709.nas.nasa.gov b0707.nas.nasa.gov b0705.nas.nasa.gov b0703.nas.nasa.gov b0615.nas.nasa.gov b0613.nas.nasa.gov b0611.nas.nasa.gov b0609.nas.nasa.gov b0607.nas.nasa.gov b0605.nas.nasa.gov b0603.nas.nasa.gov b0515.nas.nasa.gov b0513.nas.nasa.gov b0511.nas.nasa.gov b0509.nas.nasa.gov b0507.nas.nasa.gov b0505.nas.nasa.gov b0503.nas.nasa.gov ERROR: 0031-616 gethostbyname failed for home node
Jacobi Iteration - Ready send
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
rsend: 25 iterations in 0.014087 secs (1.987666 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
rsend: 25 iterations in 0.138943 secs (165.085986 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
rsend: 25 iterations in 0.017959 secs (3.118214 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
rsend: 25 iterations in 0.167202 secs (274.369185 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
rsend: 25 iterations in 0.022795 secs (4.913315 MFlops); diffnorm 0.080560, m=7 n=130 np=64
rsend: 25 iterations in 0.146721 secs (625.338290 MFlops); diffnorm 0.474303, m=4098 n=130 np=64
Jacobi Iteration - Overlapping communication
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
isend/overlap: 25 iterations in 0.014547 secs (1.924819 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
isend/overlap: 25 iterations in 0.157001 secs (146.098316 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
isend/overlap: 25 iterations in 0.017324 secs (3.232566 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
isend/overlap: 25 iterations in 0.179038 secs (256.232067 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
isend/overlap: 25 iterations in 0.022999 secs (4.869682 MFlops); diffnorm 0.080560, m=7 n=130 np=64
isend/overlap: 25 iterations in 0.180448 secs (508.459651 MFlops); diffnorm 0.474303, m=4098 n=130 np=64
Jacobi Iteration - Overlapping communication (sends first)
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
send first/overlap: 25 iterations in 0.014333 secs (1.953568 MFlops); diffnorm 0.036615, m=7 n=34 np=16
Running job under PBS
send first/overlap: 25 iterations in 0.141071 secs (162.595765 MFlops); diffnorm 0.468864, m=4098 n=34 np=16
Running job under PBS
send first/overlap: 25 iterations in 0.018143 secs (3.086586 MFlops); diffnorm 0.055291, m=7 n=66 np=32
Running job under PBS
send first/overlap: 25 iterations in 0.170800 secs (268.590714 MFlops); diffnorm 0.470684, m=4098 n=66 np=32
Running job under PBS
send first/overlap: 25 iterations in 0.025103 secs (4.461600 MFlops); diffnorm 0.080560, m=7 n=130 np=64
send first/overlap: 25 iterations in 0.176234 secs (520.617055 MFlops); diffnorm 0.474303, m=4098 n=130 np=64
Jacobi Iteration - Persistent send/recv
mpicc -c -O jacobi.c
mpicc -c -O cmdline.c
mpicc -c -O setupmesh.c
mpicc -c -O exchng.c
mpicc -o jacobi -O jacobi.o cmdline.o setupmesh.o exchng.o -lm
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
qsub failed: qsub: Job exceeds queue resource limits
mpirun aborting
p021 p020 p019 p018 p016 p015 p014 p013 p012 p008 p007 p006 p005 p004 p003 p002 Running job under PBS
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 1
ERROR: 0032-158 Persistent request already active (7) in MPI_Startall, task 0
ERROR: 0031-250 task 11: Terminated
ERROR: 0031-250 task 12: Terminated
ERROR: 0031-250 task 13: Terminated
ERROR: 0031-250 task 14: Terminated
ERROR: 0031-250 task 15: Terminated
ERROR: 0031-250 task 3: Terminated
ERROR: 0031-250 task 1: Terminated
ERROR: 0031-250 task 2: Terminated
ERROR: 0031-250 task 8: Terminated
ERROR: 0031-250 task 9: Terminated
ERROR: 0031-250 task 10: Terminated
ERROR: 0031-250 task 0: Terminated
ERROR: 0031-250 task 4: Terminated
ERROR: 0031-250 task 5: Terminated
ERROR: 0031-250 task 6: Terminated
ERROR: 0031-250 task 7: Terminated
p021 p020 p019 p018 p016 p015 p014 p013 p012 p008 p007 p006 p005 p004 p003 p002 Running job under PBS
ERROR: 0032-158 Persistent request already active (7) in MPI_Startall, task 0
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 1
ERROR: 0031-250 task 0: Terminated
ERROR: 0031-250 task 1: Terminated
ERROR: 0031-250 task 2: Terminated
ERROR: 0031-250 task 3: Terminated
ERROR: 0031-250 task 4: Terminated
ERROR: 0031-250 task 5: Terminated
ERROR: 0031-250 task 6: Terminated
ERROR: 0031-250 task 7: Terminated
ERROR: 0031-250 task 8: Terminated
ERROR: 0031-250 task 9: Terminated
ERROR: 0031-250 task 10: Terminated
ERROR: 0031-250 task 11: Terminated
ERROR: 0031-250 task 12: Terminated
ERROR: 0031-250 task 13: Terminated
ERROR: 0031-250 task 14: Terminated
ERROR: 0031-250 task 15: Terminated
Running job under PBS
ERROR: 0032-158 Persistent request already active (7) in MPI_Startall, task 1
=>> PBS: job killed: walltime 308 exceeded limit 300
ERROR: 0031-250 task 30: Terminated
ERROR: 0031-250 task 31: Terminated
ERROR: 0031-250 task 27: Terminated
ERROR: 0031-250 task 28: Terminated
ERROR: 0031-250 task 29: Terminated
ERROR: 0031-250 task 19: Terminated
ERROR: 0031-250 task 25: Terminated
ERROR: 0031-250 task 26: Terminated
ERROR: 0031-250 task 22: Terminated
ERROR: 0031-250 task 23: Terminated
ERROR: 0031-250 task 24: Terminated
ERROR: 0031-250 task 0: Terminated
ERROR: 0031-250 task 1: Terminated
ERROR: 0031-250 task 2: Terminated
ERROR: 0031-250 task 3: Terminated
ERROR: 0031-250 task 4: Terminated
ERROR: 0031-250 task 5: Terminated
ERROR: 0031-250 task 6: Terminated
ERROR: 0031-250 task 7: Terminated
ERROR: 0031-250 task 8: Terminated
ERROR: 0031-250 task 9: Terminated
ERROR: 0031-250 task 10: Terminated
ERROR: 0031-250 task 11: Terminated
ERROR: 0031-250 task 14: Terminated
ERROR: 0031-250 task 15: Terminated
ERROR: 0031-250 task 16: Terminated
ERROR: 0031-250 task 17: Terminated
ERROR: 0031-250 task 18: Terminated
ERROR: 0031-250 task 20: Terminated
ERROR: 0031-250 task 21: Terminated
ERROR: 0032-158 Persistent request already active (7) in MPI_Startall, task 1
=>> PBS: job killed: walltime 329 exceeded limit 300
sh: 77614 Terminated
ERROR: 0031-250 task 31: Terminated
ERROR: 0031-250 task 29: Terminated
ERROR: 0031-250 task 30: Terminated
ERROR: 0031-250 task 10: Terminated
ERROR: 0031-250 task 11: Terminated
ERROR: 0031-250 task 12: Terminated
ERROR: 0031-250 task 13: Terminated
ERROR: 0031-250 task 14: Terminated
ERROR: 0031-250 task 15: Terminated
ERROR: 0031-250 task 16: Terminated
ERROR: 0031-250 task 17: Terminated
ERROR: 0031-250 task 18: Terminated
ERROR: 0031-250 task 19: Terminated
ERROR: 0031-250 task 21: Terminated
ERROR: 0031-250 task 22: Terminated
ERROR: 0031-250 task 23: Terminated
ERROR: 0031-250 task 24: Terminated
ERROR: 0031-250 task 25: Terminated
ERROR: 0031-250 task 26: Terminated
ERROR: 0031-250 task 27: Terminated
ERROR: 0031-250 task 28: Terminated
ERROR: 0031-250 task 7: Terminated
ERROR: 0031-250 task 8: Terminated
ERROR: 0031-250 task 9: Terminated
ERROR: 0031-250 task 0: Terminated
ERROR: 0031-250 task 1: Terminated
ERROR: 0031-250 task 2: Terminated
ERROR: 0031-250 task 3: Terminated
ERROR: 0031-250 task 4: Terminated
ERROR: 0031-250 task 5: Terminated
ERROR: 0031-250 task 6: Terminated
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 4
ERROR: 0032-158 Persistent request already active (7) in MPI_Startall, task 2
ERROR: 0032-158 Persistent request already active (7) in MPI_Startall, task 3
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 7
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 5
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 6
=>> PBS: job killed: walltime 308 exceeded limit 300
sh: 101720 Terminated
ERROR: 0031-250 task 63: Terminated
ERROR: 0031-250 task 61: Terminated
ERROR: 0031-250 task 62: Terminated
ERROR: 0031-250 task 58: Terminated
ERROR: 0031-250 task 59: Terminated
ERROR: 0031-250 task 60: Terminated
ERROR: 0031-250 task 56: Terminated
ERROR: 0031-250 task 57: Terminated
ERROR: 0031-250 task 1: Terminated
ERROR: 0031-250 task 2: Terminated
ERROR: 0031-250 task 23: Terminated
ERROR: 0031-250 task 24: Terminated
ERROR: 0031-250 task 25: Terminated
ERROR: 0031-250 task 26: Terminated
ERROR: 0031-250 task 27: Terminated
ERROR: 0031-250 task 28: Terminated
ERROR: 0031-250 task 29: Terminated
ERROR: 0031-250 task 30: Terminated
ERROR: 0031-250 task 31: Terminated
ERROR: 0031-250 task 32: Terminated
ERROR: 0031-250 task 33: Terminated
ERROR: 0031-250 task 34: Terminated
ERROR: 0031-250 task 35: Terminated
ERROR: 0031-250 task 36: Terminated
ERROR: 0031-250 task 37: Terminated
ERROR: 0031-250 task 38: Terminated
ERROR: 0031-250 task 39: Terminated
ERROR: 0031-250 task 40: Terminated
ERROR: 0031-250 task 41: Terminated
ERROR: 0031-250 task 42: Terminated
ERROR: 0031-250 task 43: Terminated
ERROR: 0031-250 task 44: Terminated
ERROR: 0031-250 task 45: Terminated
ERROR: 0031-250 task 46: Terminated
ERROR: 0031-250 task 47: Terminated
ERROR: 0031-250 task 49: Terminated
ERROR: 0031-250 task 50: Terminated
ERROR: 0031-250 task 52: Terminated
ERROR: 0031-250 task 53: Terminated
ERROR: 0031-250 task 54: Terminated
ERROR: 0031-250 task 55: Terminated
ERROR: 0031-250 task 0: Terminated
ERROR: 0031-250 task 8: Terminated
ERROR: 0031-250 task 9: Terminated
ERROR: 0031-250 task 10: Terminated
ERROR: 0031-250 task 11: Terminated
ERROR: 0031-250 task 12: Terminated
ERROR: 0031-250 task 16: Terminated
ERROR: 0031-250 task 17: Terminated
ERROR: 0031-250 task 18: Terminated
ERROR: 0031-250 task 19: Terminated
ERROR: 0031-250 task 20: Terminated
ERROR: 0031-250 task 21: Terminated
ERROR: 0031-250 task 22: Terminated
ERROR: 0031-250 task 3: Terminated
ERROR: 0031-250 task 4: Terminated
ERROR: 0031-250 task 5: Terminated
ERROR: 0031-250 task 6: Terminated
ERROR: 0031-250 task 7: Terminated
ERROR: 0032-158 Persistent request already active (7) in MPI_Startall, task 3
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 5
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 6
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 4
ERROR: 0032-158 Persistent request already active (6) in MPI_Startall, task 7
ERROR: 0032-158 Persistent request already active (7) in MPI_Startall, task 2
=>> PBS: job killed: walltime 336 exceeded limit 300
sh: 76886 Terminated
ERROR: 0031-250 task 63: Terminated
ERROR: 0031-250 task 61: Terminated
ERROR: 0031-250 task 62: Terminated
ERROR: 0031-250 task 56: Terminated
ERROR: 0031-250 task 57: Terminated
ERROR: 0031-250 task 58: Terminated
ERROR: 0031-250 task 59: Terminated
ERROR: 0031-250 task 60: Terminated
ERROR: 0031-250 task 53: Terminated
ERROR: 0031-250 task 54: Terminated
ERROR: 0031-250 task 55: Terminated
ERROR: 0031-250 task 21: Terminated
ERROR: 0031-250 task 22: Terminated
ERROR: 0031-250 task 23: Terminated
ERROR: 0031-250 task 28: Terminated
ERROR: 0031-250 task 31: Terminated
ERROR: 0031-250 task 32: Terminated
ERROR: 0031-250 task 33: Terminated
ERROR: 0031-250 task 34: Terminated
ERROR: 0031-250 task 35: Terminated
ERROR: 0031-250 task 36: Terminated
ERROR: 0031-250 task 37: Terminated
ERROR: 0031-250 task 38: Terminated
ERROR: 0031-250 task 39: Terminated
ERROR: 0031-250 task 40: Terminated
ERROR: 0031-250 task 41: Terminated
ERROR: 0031-250 task 42: Terminated
ERROR: 0031-250 task 43: Terminated
ERROR: 0031-250 task 44: Terminated
ERROR: 0031-250 task 45: Terminated
ERROR: 0031-250 task 46: Terminated
ERROR: 0031-250 task 47: Terminated
ERROR: 0031-250 task 48: Terminated
ERROR: 0031-250 task 49: Terminated
ERROR: 0031-250 task 50: Terminated
ERROR: 0031-250 task 51: Terminated
ERROR: 0031-250 task 52: Terminated
ERROR: 0031-250 task 4: Terminated
ERROR: 0031-250 task 5: Terminated
ERROR: 0031-250 task 6: Terminated
ERROR: 0031-250 task 7: Terminated
ERROR: 0031-250 task 8: Terminated
ERROR: 0031-250 task 9: Terminated
ERROR: 0031-250 task 10: Terminated
ERROR: 0031-250 task 11: Terminated
ERROR: 0031-250 task 12: Terminated
ERROR: 0031-250 task 13: Terminated
ERROR: 0031-250 task 14: Terminated
ERROR: 0031-250 task 15: Terminated
ERROR: 0031-250 task 17: Terminated
ERROR: 0031-250 task 18: Terminated
ERROR: 0031-250 task 0: Terminated
ERROR: 0031-250 task 1: Terminated
ERROR: 0031-250 task 2: Terminated
ERROR: 0031-250 task 3: Terminated
ERROR: 0031-250 task 19: Terminated
ERROR: 0031-250 task 20: Terminated