(Talend) Stress Test 2: Flat to MySQL (RELOAD)

The Second stress test involves, Transfering 6 millions of records from a Flat File, to a table in MySQL. Version 2

The excellent timing obtained by Talend in Version 1 regarding interface 2 were shown in this new test.

Reducing the timing to 1/3, it maintains the first position compared to the rest. The tool makes use of the assigned and available resources, obtaining a remarkable performance in time, in addition to the positive aforementioned characteristics such as the native conection to MySQL.

A few changes were introduced compared to the previous configuration, Commit Size and Array Size suited to the new context (a database in a suitable environment and net), and the parameters -Xms -Xmx, according to a new hardware with 6 times more of RAM memory.



- Environment: Infraestructure composed of 3 nodes

- 1) ESXi 5.0:

1.a) Physical Datastore 1: VM ETL Clover (12GB RAM - 2 Cores * 2 Sockets)

1.b) Physical Datastore 2: VM Database Server MySQL/Oracle (4GB RAM - 2 Cores * 2 Sockets)

- 2) Monitor Performance: VM Monitor ESXi + SQL Server 2008 (with 4 GB RAM)

- 3) Operator ETL: ESXi Client (with 3 GB RAM)

CASE 1: -Xms2024M, -Xmx6096M, + 250Kcommit + 100K ArrayS


- To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file.

- Compare performances in the 2 environments.

- Analyze use of the resources

ETL Tool Talend Open Studio 5.1 (CE)
Rows: 6.024.000 M
Columns: 37 Columns



Design & Run

Elapsed time (s) 316 vs 939 Secs.
Rows p/s (avg)

VERSION 2: Performance improvement.

19.025 VS 6.415 rows/sec

How to Improve


- Adjust the parameters:

- Xms + Xmx + Csize + ArraySize (as shown in the figure above)



Important: Memory Swap: 0

CPU/Datastore: CPU Usage Mhz / Datastore usage between 23:54-00:00

Menmory: After several executions, the memory consumption remains stable in 6 GB


CPU Monitoring, "Passive and Active state" in different executions. Last Execution: 23:54-00:00


Memory Monitoring: Last Execution: 23:54-00:00