|
First stress test: Transfering 6 millions of records from a Flat File, to CSV file. |
|
It was a pleasure doing tests with tools like this, Talend and transitivity Jasper, I was amazed at key points of development. Connectivity is one of them. The processing speed with so few resources is another. Things like the logs, I did not like much, but everything can not be perfect.
Referring to this case, I played a lot with Xmx and Xms as variables, and record the changes so that they could see as impacted at runtime.
As in most of the Tests, there are several cases using different techniques, setting of variables, with the same objective, that satisfy the requirement and improve the performance. Always the first case is the best, and the rest of the cases, are jobs without tuning, or an incomplete set of techniques (but helped on the way).
From the first test we had optimal results, but try to find the point of equilibrium between the saturation of the resources and benefits.
|
LINKS
CASE 1:[183 secs.] CASE 2:[203 secs.] CASE 3:[249 secs.] |
|
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, Talend like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
|
Structure: (Metadata) |
* flexibility in the management of metadata |
| Design & Run |
|
| Elapsed time (s) | 183 Secs. |
| Rows per sec (avg) | 32.796 rows/sec |
| How to Improve Perform |
- Adjust the parameters: Xms -Xmx (as shown in the figure above) |
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, Talend like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
|
Structure: (Metadata) |
|
| Design & Run |
|
| Elapsed time (s) | 203 Secs. |
| Rows per sec (avg) | 29.651 rows/sec |
| How to Improve Perform |
- Adjust the parameters: Xms -Xmx (as shown in the figure above) |
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, Talend like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
|
Structure: (Metadata) |
* flexibility in the management of metadata
|
| Design & Run |
|
| Elapsed time (s) | 249 secs. |
| Rows per sec (avg) | 24.129 rows/sec |
| How to Improve Perform | - Adjust the parameters: Xms -Xmx (as shown in the figure above) |
