|
First stress test: Transfering 6 millions of records from a Flat File, to CSV file. |
|
This is a particular case, the successful execution of the job was limited by resources used for testing. As we have mentioned in the majority of articles, the resources we use for testing, are the minimum and necessary to run. There is no comparison with a production environment today. However, respecting the minimum requirements, and using the same technological base for all, the test is valid. There are cases like this, where the solution runs, but does not complete successfully when for example the volume of data increases.
Some of the special mention in the case of aAdeptia are :
- For all virtual machines assigned disk was X GB. In the case of Adeptia, during execution, the disk space, was consumed without passing the first stage of the job, aborting due to lack of disk space. As an exceptional measure, be reassigned twice the disk space (2X). With this modification, the execution of the job was satisfactory until the third step. (consuming 90 percent)
- The third step, "DataMapper" was aborted for exceeding the assigned heap size. The default value, was taken to the limit possible (1536M), but also was insufficient to process the file of 1GB, like all other tools.
- As the last step we only had to run the job with a cut of 1 MB file (in contrast to the 1 GB file). The run was successful. To 10 times increase file size, run the job again OK. In trying to increase the file size about 10 times more (100 MB), the job returned to an abortion.
- Below will discuss only the successful case, however will not be taken into account in the final comparison.
Later, there will be a second stage of testing with more resources, looking for new results to the cases already tested, and run those that do not run with the technological constraint imposed.
CASE 1: SOURCE FILE (15 MB.)
| Objective: | To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk. |
| Rows: | 6.024.000 M |
| Columns: | 37 Columns |
| Resources: | Virtual machine with: 2 GB RAM, Adeptia like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources) |
| File Size |
15 MB (EXCEPTION) |
| Design & Run |
|
| Elapsed time (s) | 70 Secs |
| Summary of log |
|
