|
The Second stress test involves, Pass 6 millions of records from a Flat File, to a table in MySQL. |
|
As in most of the Test, there are several cases using differents techniques, setting of variables, with the same objective, satisfy the requirement and improve the performance. Always the first case is the best, and the rest of cases, are jobs that don't have the most appropiate set up options, but helped in the way to reach the goal of improving performance.
From the first test we had optimal results because Jasper (Talend) has a specific plugin for MySQL, with great results of performance, (this differentiates it from other tools like DS,SISS,..) Equally try to find the point of equilibrium between the saturation of the resources and benefits.
|
LINKS
CASE 1:[885 secs.] CASE 2:[942 secs.] |
|
CASE 1: -Xms768M, -Xmx1536M, + 20Kcommit + 5K ArrayS
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to a MySQL database. The DB runs in another machine. The benefit is "Clear", only Jasper runs alone in the virtual machine. The negative point is the transfer of data through the network, using the generic drivers installed by VM. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, Jasper like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
|
Structure: (Metadata) |
|
| Design & Run |
|
| Elapsed time (s) | 885 Secs. |
| Rows per sec (avg) | 6.811 rows/sec |
| How to Improve Perform |
- Adjust the parameters: - Xms + Xmx + Csize + ArraySize (as shown in the figure above) |
CASE 2: -Xms512M, -Xmx1024M, + 10Kcommit + 5K ArrayS
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to a MySQL database. The DB runs in another machine. The benefit es "Clear", only Jasper runs alone in the virtual machine. The negative point is the transfer of data through the network, using the generic drivers installed by VM. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, Jasper like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
|
Structure: (Metadata) |
|
| Design & Run |
|
| Elapsed time (s) | 942 Secs. |
| Rows per sec (avg) | 6.399 rows/sec |
| How to Improve Perform |
- Adjust the parameters: - Xms + -Xmx + CSize + ArraySize (as shown in the figure above) |
