|
First stress test: Transfering 6 millions of records from a Flat File, to CSV file. |
 |
ODI's analysis began with the first test (after studying a study of architecture). As initial comment, I can say that the ease of use, not one of their characteristics. Not as intuitive as many other tools, requiring enough reading on how to interact with ODI.
3 different tests were made, varying the "Staging zone", to collect data from their different performance.
Architecture where the Staging Area is in the middle (memory) between Origin and Destination, abort. Googling to determine which was my mistake, I found out that this type of architecture only supports up to 500,000 records (possibly with boundary infrastructure like ours)
Below I show an image (which is not included in the successful cases)
As in most of the Test, there are several cases using differents techniques, setting of variables, with the same objective, satisfy the requirement and improve the performance. Always the first case is the best, and the rest of cases, are jobs without tuning, or an incomplet set of techniques (but helped on the way).
|
LINKS
CASE 1:[599 secs.] CASE 2:[1309 secs.] |
|
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, ODI like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
|
Structure: (Metadata) |
|
| Design & Run |
|
| Elapsed time (s) | 599 Secs. |
| Rows per sec (avg) | 10.050 rows/sec |
| How to Improve Perform |
- Adjust the Staging Zone |
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, ODI like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
|
Structure: (Metadata) |
|
| Design & Run |
|
| Elapsed time (s) | 1309 Secs. |
| Rows per sec (avg) | 4.600 rows/sec |
| How to Improve Perform |
- Adjust the Staging Zone |
