|
The Second stress test involves, Transfering 6 millions of records from a Flat File, to a table in MySQL. Version 2 |
|
The excellent timing obtained by Talend in Version 1 regarding interface 2 were shown in this new test.
Reducing the timing to 1/3, it maintains the first position compared to the rest. The tool makes use of the assigned and available resources, obtaining a remarkable performance in time, in addition to the positive aforementioned characteristics such as the native conection to MySQL.
A few changes were introduced compared to the previous configuration, Commit Size and Array Size suited to the new context (a database in a suitable environment and net), and the parameters -Xms -Xmx, according to a new hardware with 6 times more of RAM memory.
Times:
Architecture:
- Environment: Infraestructure composed of 3 nodes
- 1) ESXi 5.0:
1.a) Physical Datastore 1: VM ETL Clover (12GB RAM - 2 Cores * 2 Sockets)
1.b) Physical Datastore 2: VM Database Server MySQL/Oracle (4GB RAM - 2 Cores * 2 Sockets)
- 2) Monitor Performance: VM Monitor ESXi + SQL Server 2008 (with 4 GB RAM)
- 3) Operator ETL: ESXi Client (with 3 GB RAM)
CASE 1: -Xms2024M, -Xmx6096M, + 250Kcommit + 100K ArrayS
| Objective: |
- To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file. - Compare performances in the 2 environments. - Analyze use of the resources |
| ETL Tool | Talend Open Studio 5.1 (CE) |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
|
Structure: (Metadata) |
|
| Design & Run |
|
| Elapsed time (s) | 316 vs 939 Secs. |
| Rows p/s (avg) |
VERSION 2: Performance improvement. 19.025 VS 6.415 rows/sec |
|
How to Improve Performance |
- Adjust the parameters: - Xms + Xmx + Csize + ArraySize (as shown in the figure above) |
USE OF RESOURCES: VM TALEND
| TOTAL |
Important: Memory Swap: 0 CPU/Datastore: CPU Usage Mhz / Datastore usage between 23:54-00:00 Menmory: After several executions, the memory consumption remains stable in 6 GB |
| CPU: |
CPU Monitoring, "Passive and Active state" in different executions. Last Execution: 23:54-00:00
|
| Memory: |
Memory Monitoring: Last Execution: 23:54-00:00
|
| Network |
|
| DataST |
|
