(Pentaho) Stress Test 2: Flat to MySQL

The Second stress test involves, Pass 6 millions of records from a Flat File, to a table in MySQL.

DETAILS:

Objective:	To measure elapsed time reading and writing 6 million rows, from Flat file, to a MySQL database. The DB runs in another machine. The benefit is "Clear", only Pentaho runs alone in the virtual machine. The negative point is the transfer of data through the network using the generic drivers installed by VM.
Rows:	6.024.000 M
Columns:	37 Columns
Resources:	Virtual machine with: 2 GB RAM, Pentaho like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is use and to measure with the same environment (regardless of the limited resources)
Structure: (Metadata)
Design & Run
Elapsed time (s)	1h, 18 minutes, 37 seconds. (4717 sec)
Rows per sec (avg)	Here, show the evolution of the passage of rows via the transformations.
Commit	*10 K rows*
Summary of log	2012/02/29 12:02:45 - Spoon - Launching transformation [Stress2Flat_to_MySQL]... 2012/02/29 12:02:45 - Spoon - Started the transformation execution. 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Dispatching started for transformation [Stress2Flat_to_MySQL] 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Nr of arguments detected:0 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - This is not a replay transformation 2012/02/29 12:02:45 - Transformation metadata - Natural sort of steps executed in 0ms (2 time previous steps calculated) 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - I found 2 different steps to launch. 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Allocating rowsets... 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Allocating rowsets for step 0 --> DwContent_Input 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - prevcopies = 1, nextcopies=1 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Transformation allocated new rowset [DwContent_Input.0 - Table output.0] 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Allocated 1 rowsets for step 0 --> DwContent_Input 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Allocating rowsets for step 1 --> Table output 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Allocated 1 rowsets for step 1 --> Table output 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Allocating Steps & StepData... 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Transformation is about to allocate step [DwContent_Input] of type [TextFileInput] 2012/02/29 12:02:45 - DwContent_Input.0 - distribution activated 2012/02/29 12:02:45 - DwContent_Input.0 - Starting allocation of buffers & new threads... 2012/02/29 12:02:45 - DwContent_Input.0 - Step info: nrinput=0 nroutput=1 2012/02/29 12:02:45 - DwContent_Input.0 - output rel. is 1:1 2012/02/29 12:02:45 - DwContent_Input.0 - Found output rowset [DwContent_Input.0 - Table output.0] 2012/02/29 12:02:45 - DwContent_Input.0 - Finished dispatching 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Transformation has allocated a new step: [DwContent_Input].0 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Transformation is about to allocate step [Table output] of type [TableOutput] 2012/02/29 12:02:45 - Table output.0 - distribution activated 2012/02/29 12:02:45 - Table output.0 - Starting allocation of buffers & new threads... 2012/02/29 12:02:45 - Table output.0 - Step info: nrinput=1 nroutput=0 2012/02/29 12:02:45 - Table output.0 - Got previous step from [Table output] #0 --> DwContent_Input 2012/02/29 12:02:45 - Table output.0 - input rel is 1:1 2012/02/29 12:02:45 - Table output.0 - Found input rowset [DwContent_Input.0 - Table output.0] 2012/02/29 12:02:45 - Table output.0 - Finished dispatching 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Transformation has allocated a new step: [Table output].0 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - This transformation can be replayed with replay date: 2012/02/29 12:02:45 2012/02/29 12:02:45 - Stress2Flat_to_MySQL - Initialising 2 steps... 2012/02/29 12:02:45 - DwContent_Input.0 - Running on slave server #0/1. 2012/02/29 12:02:45 - Table output.0 - Running on slave server #0/1. 2012/02/29 12:02:45 - MySQLExternal - New database connection defined 2012/02/29 12:02:45 - MySQLExternal - Connected to database. 2012/02/29 12:02:45 - Table output.0 - Connected to database [MySQLExternal] (commit=10000) ----- 2012/02/29 13:09:45 - DwContent_Input.0 - linenr 5400000 2012/02/29 13:10:03 - Table output.0 - linenr 5400000 2012/02/29 13:10:40 - DwContent_Input.0 - linenr 5450000 2012/02/29 13:10:57 - Table output.0 - linenr 5450000 2012/02/29 13:11:34 - DwContent_Input.0 - linenr 5500000 2012/02/29 13:11:52 - Table output.0 - linenr 5500000 2012/02/29 13:12:28 - DwContent_Input.0 - linenr 5550000 2012/02/29 13:12:46 - Table output.0 - linenr 5550000 2012/02/29 13:13:21 - DwContent_Input.0 - linenr 5600000 2012/02/29 13:13:39 - Table output.0 - linenr 5600000 2012/02/29 13:14:16 - DwContent_Input.0 - linenr 5650000 2012/02/29 13:14:33 - Table output.0 - linenr 5650000 2012/02/29 13:15:09 - DwContent_Input.0 - linenr 5700000 2012/02/29 13:15:27 - Table output.0 - linenr 5700000 2012/02/29 13:16:02 - DwContent_Input.0 - linenr 5750000 2012/02/29 13:16:20 - Table output.0 - linenr 5750000 2012/02/29 13:16:56 - DwContent_Input.0 - linenr 5800000 2012/02/29 13:17:15 - Table output.0 - linenr 5800000 2012/02/29 13:17:51 - DwContent_Input.0 - linenr 5850000 2012/02/29 13:18:09 - Table output.0 - linenr 5850000 2012/02/29 13:18:45 - DwContent_Input.0 - linenr 5900000 2012/02/29 13:19:03 - Table output.0 - linenr 5900000 2012/02/29 13:19:40 - DwContent_Input.0 - linenr 5950000 2012/02/29 13:19:58 - Table output.0 - linenr 5950000 2012/02/29 13:20:34 - DwContent_Input.0 - linenr 6000000 2012/02/29 13:20:52 - Table output.0 - linenr 6000000 2012/02/29 13:21:06 - DwContent_Input.0 - Finished processing (I=6029427, O=0, R=0, W=6029426, U=1, E=0) 2012/02/29 13:21:22 - MySQLExternal - Connection to database closed! 2012/02/29 13:21:22 - Table output.0 - Finished processing (I=0, O=6029426, R=6029426, W=6029426, U=0, E=0) 2012/02/29 13:21:22 - Spoon - The transformation has finished!!