|
First stress test: Pass 6 millions of records from a Flat File, to CSV file. |
|
There are 3 cases, sorted ascending, being the most performant, first described. I will mention the changes applied in each case and how were variating the needle of the performance.
The goal is not to make a tutorial step by step in the development of the job, but yes mention the features and variables, to modify, to improve performance at runtime.
- Lazy Conversion
- CSV Stage
- Use Copies
- Set N|O BS
- Use Fast Data Dump
| LINKS | ||
| Case 1 | [152 Secs.] |
|
| Case 2 | [228 Secs.] | |
| Case 3 | [384 Secs.] | |
| Case 4 | [715 Secs.] | |
CASE 1: CSV + Lazy Conv + X4 Cop + FData dump + N|O BS 1.500.000 (Best)
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Resources: | Virtual machine with: 2 GB RAM, Clover like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
| Structure Metadata: |  |
| Design & Run |
|
| Elapsed time (s) | 152 sec. |
| Rows per sec (avg) |
Here, show the evolution of the passage of rows via the transformations
|
| Improve Perform. |
- Use CSV - - Use Lazy Conversion - Use Fast Data Dump - Set N|O BS to 1.5M - Set 4X (Copies) |
CASE 2: CSV + Lazy Conv + X3 Cop + FData dump + N|O BS 80.000 (2nd Place)
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Resources: | Virtual machine with: 2 GB RAM, Clover like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
| Structure Metadata: | |
| Design & Run |
|
| Elapsed time (s) | 228 sec. |
| Rows per sec (avg) |
Here, show the evolution of the passage of rows via the transformations
|
| Improve Perform. |
- Use CSV - - Use Lazy Conversion - Use Fast Data Dump - Set N|O BS to 80.000 - Set 3X (Copies) |
CASE 3: CSV + Lazy Conv + X2 Cop + FData dump + N|O BS 50.000 (3rd Place)
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Resources: | Virtual machine with: 2 GB RAM, Clover like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
| Structure Metadata: | |
| Design & Run |
|
| Elapsed time (s) | 384 sec. |
| Rows per sec (avg) |
Here, show the evolution of the passage of rows via the transformations
|
| Improve Perform. |
- Use CSV - - Use Lazy Conversion - Use Fast Data Dump - Set N|O BS to 50.000 - Set 2X (Copies) |
CASE 4: TextInput Stage (Worst)
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Resources: | Virtual machine with: 2 GB RAM, Clover like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
| Structure Metadata: | |
| Design & Run |
|
| Elapsed time (s) | 715 sec. |
| Rows per sec (avg) |
Here, show the evolution of the passage of rows via the transformations
|
