|
Another analysis to do is measure the time taken for the transformation of "lookup" and "filter" with the same file that the previous tests. |
 |
From the first test we had optimal results. Equally try to find the point of equilibrium between the saturation of the resources and benefits. When modifying the parameters, the changes were not significant, so they were not recorded.
Between all combinations of adjustable parameters, the results were around 59 secs( the best) to 80. All combinations are recorded in case someone needs to document a case. (for reasons of time and redundancy are not exposed in this article, but let me know if you need and we will gladly send them).
CASE 1: -Xms256M, -Xmx1536M
| Objective: | To measure elapsed time reading 6 million rows, from Flat file, join the main flow with a lookup table (MySql) and take attributes. Filter the flow and write a txt file. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, Jasper like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
|
Structure: (Metadata) |
|
| Design & Run |
Join & Take attributes
Setting MySQL Connection
Filter the data:
|
| Elapsed time (s) | 59 Secs. |
| Rows per sec (avg) | 100.515 rows/sec |
| How to Improve Perform |
- Adjust the parameters: - Xms + Xmx (as shown in the figure above) |
