|
The Second stress test involves Transfering 6 millions of records from a Flat File, to a table in MySQL. Version 2 |
|
The native conection to MySQL in Clover, put together with a suitable combination of Array Size and Commit Size, plus the availability of resources and a net without traffic obtains acceptable timing, much better than the ones in Version 1 (1/3 compared to V1). It would be interesting to measure the Job changing the UniversalReader to ParallelReader, not available in CE version. (A test version with some TRIAL version FULL with Parallel will be uploaded soon).
In most of the articles there is no Step By Step explanation of the development of the experience but a highlighting of the most interesting applied configurations, so based on them, we perform an analysis of the resources. Having said this, attached you will find a reference to Clover's blog, where Michal performs an experience from connection to MySQL from scratch.
http://blog.cloveretl.com/author/michal-tomcanyi
Times:
Architecture:
- Environment: Infraestructure composed of 3 nodes
- 1) ESXi 5.0:
1.a) Physical Datastore 1: VM ETL Clover (12GB RAM - 2 Cores * 2 Sockets)
1.b) Physical Datastore 2: VM Database Server MySQL/Oracle (4GB RAM - 2 Cores * 2 Sockets)
- 2) Monitor Performance: VM Monitor ESXi + SQL Server 2008 (with 4 GB RAM)
- 3) Operator ETL: ESXi Client (with 3 GB RAM)
CASE 1: -XMX: 7144 + Array Size+ Commit Size: 250.000
| Objective: |
- To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file. - Compare performances in the 2 environments. - Analyze use of the resources |
| ETL Tool | Clover 3.3.0 (CE) |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
|
Structure: (Metadata) |
|
| Design & Run |
Run Configuration:
|
| LOG |
Log:
|
| Elapsed time (s) | 1153 Secs. |
|
How to Improve Performance |
-Adjust the parameters: COMMIT SIZE / XMX / ARRAY SIZE |
USE OF RESOURCES:VM CLOVER
| TOTAL |
Important: MemorySwap: 0 CPU/Datastore: CPU Usage Mhz / Datastore usage between 21:47-22:07 Menmory:After several executions, the memory consumption remains stable in 3,8 GB
|
| CPU: |
CPU Monitoring, "Passive and Active state" in different executions.Last Execution: 21:47-22:07
|
| Memory: |
Memory Monitoring: Last Execution: 21:47-22:06
|
| Network |
|
| DataST |
|
