|
First stress test: Transfering 6 millions of records from a Flat File, to CSV file. Version RELOAD, with a new Architecture |
 |
In this new article, will replicate the Stress Test (ODI) Stress Test 1: Flat to Flat using the new hardware architecture. The "logic model" that has been implemented is the solution with best performance of previous version. The time in the "domestic version" of this test was 599 seconds. This time, the magnitude of Hardware is Productive, expecting increase the productivity curve, and consequently, the times should decrease considerably.
The result of the experience was 77 seconds. Clearly, with the reallocation of resources we again noticed a big change in Performance. Still, it is far from tools like Pentaho (Pentaho) Stress Test 1: Flat to Flat (RELOAD), (always focused on this particular test).
| Concerning to the following error found in the first executions: "Architecture where the Staging Area is in the middle (memory) between Origin and Destination, abort. Googling to determine which was my mistake, I found out that this type of architecture only supports up to 500,000 records (possibly with boundary infrastructure like ours)" is recurrent even with more resources allocated. That is why the Interface configuration was: |  |
ARCHITECTURE:
|
Environment: Infraestructure composed of 3 nodes 1) ESXi 5.0: a) Physical Datastore 1: VM ETL ODI (12GB RAM - 2 Cores * 2 Sockets) b) Physical Datastore 2: VM Database Server Oracle (4GB RAM - 2 Cores * 2 Sockets) 2) Monitor Performance: VM Monitor ESXi + SQL Server 2008 (with 4 GB RAM) 3) Operator ETL: ESXi Client (with 4 GB RAM) |
 |
CASE 1:
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| ETL Tool: | Oracle Data Integrator 11g |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
|
Structure: (Metadata) |
|
| Design & Run |
|
| Elapsed time (s) | 77 Secs. |
| Rows per sec (avg) | 78.233 VS 10.050 rows/sec |
| How to Improve Perform |
- Adjust the Staging Zone |
USE OF RESOURCES: VM ORACLE DATA INTEGRATOR 11g
| CPU: |
CPU Monitoring, "Passive and Active state" in different executions. Last Execution: 20:53-20:55
Performance chart legend |
| Memory: |
Memory Monitoring: Last Execution: 20:53-20:55
Performance chart legend |
| DataST |
Performance chart legend |
