|
First stress test: Transfering 6 millions of records from a Flat File, to CSV file.V2 |
|
This is the first test that replied in the new environment. It's very satisfying to be able to run a "stress test" without having to worry about resources. In the particular case of Talend, it was possible to increase the "JVM settings", with respect to the version 1.
Memory consumption reached nearly 6 GB of memory, without "memory swaped". The time was 4 times less than the first version.
Architecture:
- Environment: Infraestructure composed of 3 nodes
- 1) ESXi 5.0:
1.a) Physical Datastore 1: VM ETL TALEND (12GB RAM - 2 Cores * 2 Sockets)
1.b) Physical Datastore 2: VM Database Server MySQL/Oracle (4GB RAM - 2 Cores * 2 Sockets)
- 2) Monitor Performance: VM Monitor ESXi + SQL Server 2008 (with 4 GB RAM)
- 3) Operator ETL: ESXi Client (with 3 GB RAM)
CASE 1: -Xms1024M, -Xmx3096M
| Objective: |
- To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. - Compare performances in the 2 environments. - Analyze use of the resources |
| ETL Tool | Talend Open Studio 5.1 (CE) |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
|
Structure: (Metadata) |
* flexibility in the management of metadata |
| Design & Run |
|
| Elapsed time (s) | 45 Secs. |
| Rows p/s (avg) | 133.939 rows/sec VS 32.796 ros/Sec (Test1_V1) |
| How to Improve Perform |
- Adjust the parameters: Xms -Xmx (as shown in the figure above) |
USE OF RESOURCES:
| TOTAL |
Important: Memory Swap: 0 / Network usage: 0 CPU/Datastore: CPU Usage Mhz / Datastore usage between 0:32-0:34 y 0:40-042 Menmory: After several executions, the memory consumption remains stable in 6 GB |
| CPU: |
CPU Monitoring, "Passive and Active state" in different executions.
|
| Memory: |
Memory Monitoring |
