Customised ETL Solution for Client in Qatar
We were engaged along with a large Indian System Integrator to replace an ETL solution which was unacceptably inefficient for their Data Warehouse on Oracle platform.
The ETL – Extract, Transform and Load – populated Data Warehouse consisting of electricity and water billing transactions and financial accounting data at summary level.
The original solution using Data Stage took about 12 hours every night to incrementally refresh Data Warehouse. Often, the ETL job was not finished before the next day’s operations began.
We addressed this challenge and built an ETL solution using Oracle DI tools. The new ETL solution takes about 3 hours time to refresh Data warehouse every night.
One significant challenge that we dealt with was lack of documentation; so we reverse engineered the solution:
- We could not make any changes in Data Warehouse structure as a huge number of reports were using them
- Huge number of data elements were mapped between JD Edwards ERP system and Data Warehouse
- Hundreds of data transformation rules were decoded from the existing system and specifications were written
- Data tally was a critical part of the project delivery with certification of 100 % tally of about 80 selected reports.
We provided complete documentation, training and six months of support before handing over the solution to the local IT team.