The Problems with Siloed Databases Part 2

I received several comments on my last post, Please, Stop Creating Microsoft Access Databases and thought I'd use today's post to respond to some of them.

  • "It's more often poor planning, lack of knowledge transfer and / or changes of use over time." - I completely agree that this is what makes ongoing database and application support complex. I accept all of these as reality so the question is, are non-developers or business users developing databases with structures and documentation that simplify changes? If they are performing a one time data analysis, then maybe this isn't a concern but for databases that will be used and updated over time, they should be managed by database developers and dba's that are trained to support enhancements and changes.

  • "The underlying problem here isn't MS Access." - MS Access is unique in that it is widely available to business users, it easily allows saving databases to desktop hard drives that are difficult to administer, and has application functionality such as forms and reports. So yes, one of the underlying problems is MS Access because of its capabilities and how it is deployed.

  • "Can we arrange a process of promotion, where ad hoc dbs get promoted to proper data in due course?" - Yes, this is possible and ideal, but hard to govern and sometimes difficult to staff. It depends on how much database development is in practice and the size of the organization. My policy would read something like:
    • Register all non-IT database development in a directory.
    • Allow databases to be created for one time data analysis, but archive them in three months or less.
    • Prototype databases for single user use, but if multiple users need access or if form/reports are needed, then the prototype should be transferred to IT so that they can be properly developed and managed. 

  • "Non-developers create Access databases because they need to get some work done" - I agree, and relying on IT isn't always the answer. However, most non-developers don't have an objective to create a database - they are usually looking to develop a workflow or to perform some analytics or reporting and realize they need a database to store the data. To that end, I think it is better for IT organizations to provide "self service" tools to manage departmental workflows (see my post: In my CIO toolkit), or tools for self service analytics/reporting.

  • "It turns out it is about dark data and how organizations should better consider their enterprise data handling." "Dark data can be a problem." - Indeed, that is really what the post is about. My definition of dark data is "Data that isn't documented or easily understood, data that can't easily be connected to other data sources, or data that can't easily be used in analytics.". So when you have poorly planned, silo'ed databases, then this is a dark data issue.

5 comments:

  1. MS Access Post from 2003: Beware the Rogue ASP

    Our guideline is to not use Access for more than analysis, but unfortunately, people will take 'analysis' and stretch that," the CIO says.

    ReplyDelete
    Replies
    1. I tried coming to your defense in the initial post, where the dissenters were surprisingly nasty and supporters made great points (I tend to agree about the post title having qualifiers, like "Stop Using Access Unless You Are a DBA OR, Failing That, You Intend To Delete The Locally Made DB After You've Accomplished The Task." Just less blanket condemnation. Anyway the reason I'm even taking time to comment now is because I think my Google search string that brought up your post is a great hilariously apt real world example underpinning your article: "best way to take over someones former messy access database and vba reporting". Hahaha I was merely looking for some efficiency tips, as I am currently the sole analyst tasked with making sense of/continuing the reporting of a recently deceased database guru whose myriad databases and VBA processes--the latter in both Access and Excel--are one unending labyrinth of crappy conventions, failure to delete or archive prior bad tables/query attempts, lack of compartmentalization (so the Access and Excel files have more data sets and do more tasks than a person who cares about others' understandibg would EVER have implemented.

      I keep having to figuratively come up for air so I don't drown in the chaos (meaning I literally inch toward panic regularly throughout this untangling process).

      Starting from scratch has often been my approach here, but presently I fear I must salvage much of the former process in order to get results for my boss on time... Sigh.

      Delete
  2. Yeah its a good article. According to you what we project managers do is communicating. And a lot of this communication is done during project meetings. It can sometimes feel like you are running from one meeting to another and that your time is often wasted. Meetings don’t start on time, the issues aren’t dealt with, there is no agenda, there is no focus, nobody assigns any follow ups or tasks and of course then they also don’t end on time. An efficient project manager is required for the good management of a project. I think a project manager should PMP certified. Looking forwards to apply what I learned in PMP classes in my company.

    ReplyDelete
  3. The problem is not Microsoft Access. The problem is management and/or the IT department.

    Okay, you get a job to fix a database that is not a text book designed system. Get over it, you have a job for the next 6 - 12 months (obviously you would account for the issues in your quote, might even be a little open-ended). Would you rather a job for the next 6 - 12 months or waiting for a job?

    Surely a competent programmer/analyst could work through a mess, yes, there maybe additional costs for time but that's management's problem not the person initially creating the Access database.

    ReplyDelete
  4. Thanks for the article- definitely something to think about. I would have loved to have seen a paragraph at the end of the first article "Please Stop Creating Microsoft Access Databases" with suggestions for what to do instead. (On an unrelated note, it would also be helpful to see dates instead of just times, so that I know how old this article is!) Thanks

    ReplyDelete

Share