The Database Decision that Will Hurt Your Company's Big Data Opportunities

One year ago, I completed the controversial post, Please Stop Creating Microsoft Access Databases. It drew an onslaught of comments from those that have been successful developing on this platform and others who struggle with the issues caused by siloed databases. Now, one year later I've decided to revisit this topic.

Developers Love their Platforms - But Big Data Platforms Are Strategic


Developers tend to be passionate about the platforms that they are most knowledgeable on and have had the most success. Fifteen years ago when there were fewer mainstream platforms, there were significant debates on Java versus .Net versus ColdFusion or MySQL versus Oracle versus Microsoft. It was rare to find someone who was technically proficient and objective about these platforms and in most situations, one could substitute one platform for the other, leverage its strengths and compensate for its weaknesses. The scope for selecting a platform was largely around application development, maintenance, scalability and performance.

But I maintain that databases today are a different and changed domain. Their importance far exceeds internal and external application development needs - database platforms are the foundation for most corporations' big data, analytics, and data driven practices. The schemas developed will likely be connected to other databases and leveraged for many needs beyond their original purposes. They will most definitely be maintained and extended by developers that didn't author them, and ideally they will be interfaced by business users leveraging self-service tools to perform analytics.

Database technologies have also changed significantly including the more mainstream adoption of NoSQL and Big Data platforms. We now debate the virtues of key value stores, tuple spaces, xml repositories, columnar databases, and graph databases. Larger scaled relational databases also include some of these capabilities and provide development tools from rapid and simple to enterprise or complex.

Database Platforms are Strategic Investments


So, while it is true that many developers can learn to do a lot with Microsoft Access, the question I ask is, why? Why stick with a platform that was largely designed for desktop data stores when there are cloud databases offering similar RAD capabilities and significantly better scalability? Why stick with a database approach that has limited tools to connect, develop relationships, and share information with other data sources? Why limit your organizations to 2GB and other physical limitations?

Mind you, that I am usually support RAD and simple development tools - tools that get things done quickly and easily. But developers need to have a target reference architecture and a target data model, and I fail to see how a collection of MS Access databases is a smart long term play when connected, transparent, and quality data sources are so key to business.

If you see your role as database developer, perhaps you should have higher aspirations. You create and maintain the databases, but you should be (or aspire to be) a data technology expert and a subject matter expert on the data, its data quality, and its analytics potential. I would suggest thinking broadly about your role and the technologies where you develop your expertise. Look beyond getting today's needs 'done'.

 

8 comments:

  1. I like using Access for its simplicity. But nowadays I extract a subset of data from a more robust database so I can run quick queries and create charts and graphs. But agreed its not by any means a Big Data solution!

    ReplyDelete
    Replies
    1. Thanks for the comment! There are many lightweight tools that will allow you to easily query/filter data and create charts/graphs directly to your database. No need to extract a data subset!

      Delete
  2. Dear Isaac. Your reply to jollyrogerfl above simply shows how much hatred and bias you have for msaccess. Access is meant for a purpose and it fulfills that purpose very well. No one here has said access should be used to build a mainstream banking application to handles millions of transactions per minute.

    I have built a Payroll & HR solution that is used by most outsourcing companies here in my country Nigeria using Access. My solution processes the payroll of 25,000 employees in less than 5 minutes and gets all the computations done and reports ready. The question is how many organisations have more than 25,000 employees in the world. In essence my solution can handle the payroll of even some of the largest corporations in the world. Please use Access wisely and for the purpose it is meant. Thank you.

    ReplyDelete
  3. I am a bit late finding this post, like a year late, but I am going to submit my comment anyway so that people who stumble across this site like I did are not misled by this nonsense.

    To start with, you are approaching this from a flawed understanding of successful businesses. All IT people nowadays seem to think that you're just better off investing huge sums of money to "do things right" or to save the up front investment and go with a SaaS solution. This often requires taking a "revenue stream" approach to business rather than a "reinvestment of capital" approach. Unfortunately, that whole business model is a terrible idea for most businesses. Apparently America is going to have to learn this lesson the hard way. Additionally, it is extremely important for small businesses to have complete ownership of their tools, because the need to cut expenses may be right around the corner, and the less fixed costs you have, the better chance you have to survive the rough times.

    So, with that said, contrary to what you claim there are very few, if any, "lightweight" tools that will allow you to accomplish what Access can. Most of the tools available today rely upon layers and layers of software, APIs, web servers, database servers, frameworks, and the list goes on and on. Plus it all has to be configured correctly. This is not what most people, especially those with ZERO IT staff, and very little extra cash, consider to be "lightweight."

    So what is it that Access can do that other tools can't?

    To start with, there is one thing that Access can do that very very VERY few other applications can do...and that is link to data tables from huge variety of different sources and allow you to write SQL statements joining those tables without jumping through any hoops. I have Access apps that link to PostgreSQL, ProvideX (database for Sage 100 ERP), and SQL Server and use the data as if it were in one database. SQL Server has linked tables, but there are limitations and you cannot always easily refer to fields in those tables easily.

    In addition, you can then use those tables or resulting views in databound forms and controls. This makes it great for creating front ends for specific business tasks that follow business processes which companies are unable or sometimes unwilling to change to adapt to software restrictions. It is SIGNIFICANTLY easier for business people to develop forms that are bound than it is to populate unbound forms with data at runtime.

    You can then provide access to these tools and this data at MUCH MUCH lower licensing costs and FAR easier deployment. You don't get much easier than throwing an MDB file up on a terminal server.

    So, you clearly live in a very different world. You clearly are very immersed in I.T., but disconnected from the rest of the business world. That disconnection is why I.T. is such a risk for so many unsuspecting small companies.

    ReplyDelete
    Replies
    1. Matthew - Thanks for the reply. I realize that some developers might have alternate views on this technology and approach to database development.

      What I can tell you definitively is that when organizations rely on developing problem specific Access db's over a long period of time, it makes it very difficult to unwind and create a more holistic data strategy. Unfortunately, I've seen it too often and what looks like simple investments to solve a business problem becomes difficult to work with especially for businesses where data assets are critical. Over time, linking databases actually makes the problem worse because it's hard for others to understand all the dependencies between databases and data sources.

      There are many lightweight alternatives - mostly SaaS and PaaS solutions that compete with Access. They too have issues if you make too many dbs that are point solutions or have other inherit complexities, but offer the same RAD (and often better) capabilities of Access. Have a look at Quickbase and Force.com and the link below for additional details

      http://www.nten.org/articles/2012/do-it-yourself-cloud-databases

      Also, this post on my blog might interest you

      http://blogs.starcio.com/2014/07/paas-cloud-agility.html

      Delete
  4. Isaac,
    You obviously have not learned from the comments in your previous blog. Many people gave you great insight and time to train you.
    Your response in this blog shows you are narrow-minded and that makes you a bit dangerous in blogging.
    I follow good thinking and your response above is not good thinking or respectful to those that spent a great deal of time responding to your first post.
    I am disappointed and no longer will follow you.
    Good bye.

    ReplyDelete
    Replies
    1. Sorry to disapoint you. Not sure what comments you are referring to specifically, but I expect and publish those who have differences in opinion.

      These posts are about establishing a data strategy that often requires organizations to connect and develop data relationships between their core entities and sources. Harder to do this if developers are still building single purpose databases for single apps, even if links are used.

      There are plenty of times when a single db is appropriate, but for those situations I recommend looking at Cloud/SaaS db platforms as a service.

      Delete

Comments on this blog are moderated and we do not accept comments that have links to other websites.

Share