I recently read a blog post and two papers by Barry Devlin in which he considers whether it’s time for some major evolution in data warehouse architecture. His point is that our “venerable” data warehouse architecture has evolved over decades with a set of assumptions about database hardware and software. In a recent blog post on the BeyeNetwork site entitled Data Warehouses and Solid State Disks (SSD) , Devlin says
“Over the past couple of years, we’ve seen dramatic improvements in database performance due to hardware and software advances such as in-memory databases, columnar storage, massively parallel processing, compression, and so on …”.
He goes on to question several of the old assumptions about what is possible in a single database. The most interesting one for me is, as he puts it:
“Do you still need that Data Mart? With so much faster performance, maybe the queries you now run in the Mart could run directly on the EDW. Reducing data duplication has enormous benefits, on storage volumes, but principally in reducing maintenance of ETL to the Marts.”
I have no experience with the new technologies that he cites. But I have lots of experience with today’s established DW architectures. If an investment in new technology could significantly reduce the amount and complexity of data warehouse processing, then it’s worth serious consideration.
I’m just beginning to draft a road map for a real BI/DW program for my new employer. I’m ready to lead the BI/DW team in a charge down well-trodden paths of design and implementation. I’m formulating resource plans for building out the team. This is what they hired me to do. They’re paying for my experience in doing these things. Now I’m wondering if at least to some extent, I would be doing a disservice to the company by employing that experience.
This is exactly the kind situation that got me interested in documenting my experiences in this job – a blank slate is both compelling and intimidating. I don’t have a lot of answers at the moment. But I’m liking the questions.