|
Description of the first stress test: Transfer 6 millions of records from a Flat File, to CSV file. V2 with more resources. |
|
In this first test of "Clover" with the available resources of the new architecture, an improvement in times of almost 80 percent was obtained in comparison to version 1. Memory consumption reached nearly 4.5 GB of memory. The disk is the most important factor in this improvement
Architecture:
- Environment: Infraestructure composed of 3 nodes
- 1) ESXi 5.0:
1.a) Physical Datastore 1: VM ETL Clover (12GB RAM - 2 Cores * 2 Sockets)
1.b) Physical Datastore 2: VM Database Server MySQL/Oracle (4GB RAM - 2 Cores * 2 Sockets)
- 2) Monitor Performance: VM Monitor ESXi + SQL Server 2008 (with 4 GB RAM)
- 3) Operator ETL: ESXi Client (with 3 GB RAM)
CASE 1: -Xmx6.144MB
| Objective: |
- To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file. - Compare performances in the 2 environments. - Analyze use of the resources |
| ETL Tool | Clover 3.3.0 (CE) |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
|
Structure: (Metadata) |
* flexibility in the management of metadata |
| Design & Run |
|
| Elapsed time (s) | 39 Secs. |
| Rows p/s (avg) |
150.000 rows/sec VS 27.658 ros/Sec (Test1_V1)
|
|
How to Improve Performance |
- Adjust the parameters: -Xmx (as shown in the figure above) |
USE OF RESOURCES:
| TOTAL |
Important: Memory Swap: 0 / Network usage: 0 CPU/Datastore: CPU Usage Mhz / Datastore usage between 14:17-14:19 Menmory: After several executions, the memory consumption remains stable in 4,3 GB |
| CPU: |
CPU Monitoring, "Passive and Active state" in different executions. Last Execution:14:17 - 14:19
|
| Memory: |
Memory Monitoring: Last Execution:14:17 - 14:19
|
| DataST |
|
