|
Description of the first stress test: Transfering 6 millions of records from a Flat File, to CSV file. |
|
As in most of the Test, there are several cases using differents techniques, setting of variables, with the same objective, satisfy the requirement and improve the performance. Always the first case is the best, and the rest of cases, are jobs without tuning, or an incomplet set of techniques (but helped on the way).
|
LINKS
|
|
CASE 1: -ServerMode, -Xmx = 1500MB
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, Clover like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
| Structure: |  |
| Design & Run |  |
| Elapsed time (s) | 214 Secs. |
| Rows per sec (avg) |
Here, show the evolution of the passage of rows via the transformations, each 5 seconds.
27.658 rows/sec
|
| How to Improve Perform |
Use -server mode Set -Xmx (1599 MB)
|
| Summary of log | INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 RUNNING INFO [WatchDog] - %cpu:0.37 Out:0 5271957 1340530 27622 7165 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 RUNNING INFO [WatchDog] - %cpu:0.50 In:0 5271957 1340530 27622 7165 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **------------------- INFO [WatchDog] - Time: 08/03/12 11:55:13 INFO [WatchDog] - Node ID Port #Records #KB Rec/s KB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 RUNNING INFO [WatchDog] - %cpu:0.37 Out:0 5418730 1377426 29261 7355 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 RUNNING INFO [WatchDog] - %cpu:0.50 In:0 5418730 1377426 29261 7355 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **------------------- INFO [WatchDog] - Time: 08/03/12 11:55:18 INFO [WatchDog] - Node ID Port #Records #KB Rec/s KB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 RUNNING INFO [WatchDog] - %cpu:0.37 Out:0 5557073 1412188 27575 6928 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 RUNNING INFO [WatchDog] - %cpu:0.50 In:0 5557073 1412188 27574 6928 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **------------------- INFO [WatchDog] - Time: 08/03/12 11:55:23 INFO [WatchDog] - Node ID Port #Records #KB Rec/s KB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 RUNNING INFO [WatchDog] - %cpu:0.37 Out:0 5697879 1447585 28100 7064 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 RUNNING INFO [WatchDog] - %cpu:0.50 In:0 5697879 1447585 28100 7064 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **------------------- INFO [WatchDog] - Time: 08/03/12 11:55:28 INFO [WatchDog] - Node ID Port #Records #KB Rec/s KB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 RUNNING INFO [WatchDog] - %cpu:0.37 Out:0 5828244 1480337 25913 6510 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 RUNNING INFO [WatchDog] - %cpu:0.50 In:0 5828244 1480337 25913 6510 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **------------------- INFO [WatchDog] - Time: 08/03/12 11:55:33 INFO [WatchDog] - Node ID Port #Records #KB Rec/s KB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 RUNNING INFO [WatchDog] - %cpu:0.36 Out:0 5959535 1513343 26165 6578 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 RUNNING INFO [WatchDog] - %cpu:0.50 In:0 5959535 1513343 26165 6578 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - [Clover] Post-execute phase finalization: 0 INFO [WatchDog] - [Clover] phase: 0 post-execute finalization successfully. INFO [WatchDog] - ----------------------** Final tracking Log for phase [0] **--------------------- INFO [WatchDog] - Time: 08/03/12 11:55:37 INFO [WatchDog] - Node ID Port #Records #KB aRec/s aKB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 FINISHED_OK INFO [WatchDog] - %cpu:0.38 Out:0 6029426 1530895 28174 7153 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 FINISHED_OK INFO [WatchDog] - %cpu:0.51 In:0 6029426 1530895 28174 7153 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - Execution of phase [0] successfully finished - elapsed time(sec): 214 INFO [WatchDog] - -----------------------** Summary of Phases execution **--------------------- INFO [WatchDog] - Phase# Finished Status RunTime(sec) MemoryAllocation(KB) INFO [WatchDog] - 0 FINISHED_OK 214 12591 INFO [WatchDog] - ------------------------------** End of Summary **--------------------------- INFO [WatchDog] - WatchDog thread finished - total execution time: 214 (sec) INFO [main] - Freeing graph resources. INFO [main] - Execution of graph successful ! |
CASE 2: Without -ServerMode, -Xmx = 0
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, Clover like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
| Structure: | |
| Design & Run |  |
| Elapsed time (s) | 503 sec. |
| Rows per sec (avg) |
Here, show the evolution of the passage of rows via the transformations, each 5 seconds.
12.101 rows/sec
|
| % cpu |
Node reader: 0,18 % a 0,34% Node writer: 0,18% a 0,50%
|
| Summary of log | INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **------------------- INFO [WatchDog] - Time: 22/02/12 15:10:29 INFO [WatchDog] - Node ID Port #Records #KB Rec/s KB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 RUNNING INFO [WatchDog] - %cpu:0.27 Out:0 5943377 1509281 12676 3187 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 RUNNING INFO [WatchDog] - %cpu:0.41 In:0 5943377 1509281 12676 3187 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **------------------- INFO [WatchDog] - Time: 22/02/12 15:10:34 INFO [WatchDog] - Node ID Port #Records #KB Rec/s KB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 RUNNING INFO [WatchDog] - %cpu:0.27 Out:0 5999961 1523499 11250 2826 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 RUNNING INFO [WatchDog] - %cpu:0.41 In:0 5999961 1523499 11250 2826 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - ---------------------** Start of tracking Log for phase [0] **------------------- INFO [WatchDog] - Time: 22/02/12 15:10:39 INFO [WatchDog] - Node ID Port #Records #KB Rec/s KB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 RUNNING INFO [WatchDog] - %cpu:0.27 Out:0 6023226 1529339 4638 1164 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 RUNNING INFO [WatchDog] - %cpu:0.41 In:0 6023226 1529339 4638 1164 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - [Clover] Post-execute phase finalization: 0 INFO [WatchDog] - [Clover] phase: 0 post-execute finalization successfully. INFO [WatchDog] - ----------------------** Final tracking Log for phase [0] **--------------------- INFO [WatchDog] - Time: 22/02/12 15:10:40 INFO [WatchDog] - Node ID Port #Records #KB aRec/s aKB/s INFO [WatchDog] - --------------------------------------------------------------------------------- INFO [WatchDog] - UniversalDataReader DATA_READER0 FINISHED_OK INFO [WatchDog] - %cpu:0.34 Out:0 6029426 1530895 11986 3043 INFO [WatchDog] - UniversalDataWriter DATA_WRITER0 FINISHED_OK INFO [WatchDog] - %cpu:0.50 In:0 6029426 1530895 11986 3043 INFO [WatchDog] - ---------------------------------** End of Log **-------------------------------- INFO [WatchDog] - Execution of phase [0] successfully finished - elapsed time(sec): 503 INFO [WatchDog] - -----------------------** Summary of Phases execution **--------------------- INFO [WatchDog] - Phase# Finished Status RunTime(sec) MemoryAllocation(KB) INFO [WatchDog] - 0 FINISHED_OK 503 9152 INFO [WatchDog] - ------------------------------** End of Summary **--------------------------- INFO [WatchDog] - WatchDog thread finished - total execution time: 503 (sec) INFO [main] - Freeing graph resources. INFO [main] - Execution of graph successful ! |
