Wednesday, November 19, 2014

How Artificial Intelligence Will Solve IoT's Big Data Challenges

If IoT is going to deliver on its transformational promise, it will have to provide greater value and importance than a single internet enabled sensor such as a wearable device. The technology to create a central hub around a small collection of sensors, for example in home automation, has been around for decades. What is revolutionary today is that home automation is cheaper to implement and gives home owners better software to monitor and control their homes remotely.

As I've said in a previous post, the real magic happens when a hub of managed sensors can easily communicate with other neighboring systems. Each hub has to be programmed to intelligently broadcast signals to its neighbors and also make intelligent decisions on how to process signals from its neighbors. 

Examples of IoT Magic

Here are some examples. Maybe there are heavy rains and my basement is beginning to take on water. Can my home automation system alert neighboring systems that flooding is eminent and that the home owner should consider precautionary measures? What if my fire alarms also made similar communications if there is an emergency? These are all examples of neighboring systems sharing information.

Information sharing scales regionally as well. What if the town's facility's team was alerted when multiple homes on a block were flooding. They could then come on sight, review the environment for issues, and maybe fix sewer drains that were flooding. What if fire departments received information on the severity of a fire and its location before a 911 call?

The Challenges of Information Sharing

Information sharing has two fundamental starting problems. First, the security and privacy of what information is shared needs definition. What information and under what circumstances am I willing to share with neighbors and the extended region? It's hard enough for most users to set their Facebook privacy settings, so home automation and wearable device manufacturers will need to consider how to simplify selecting these preferences.

Then there is the question of how various systems will process a larger scope of data coming from neighboring and regional systems. Now for small scale systems with well understood patterns, a rule based approach implementing "if this then that" may be sufficient. But for ecosystems with millions of sensors, thousands of neighboring systems and hundreds of regional systems, rule based systems are likely to be too complex to define and program - assuming patterns are well understood.

Where AI Meets IoT

AI bases algorithms have a better chance to succeed in places where rule based systems are too complex to program or need to process too much data. Neural networks identifying patterns, fuzzy logic based controllers that can respond to local, neighboring and regional inputs, reinforcing learning algorithms instituted at regional systems to identify macro conditions are all AI possibilities to help transform dumb internet connected sensors to intelligent ecosystems.

AI could be used to determine "dangerous" conditions when local systems may be permissioned to "share" additional information with neighbors and regions. For example, upon sensing a flooding danger, a neighbor's home automation may proactively turn on the basement sump pump in response. A health monitoring wearable device could be programmed to seek out a nearby and volunteering doctor, medic or nurse on an emergency condition.

These are all interesting and promising AI applications in IoT. The trick will be in getting enough participants and early adopters to establish data sets, test user interfaces, and validate AI's logic and response.

continue reading "How Artificial Intelligence Will Solve IoT's Big Data Challenges"

Thursday, November 13, 2014

The Death of Microsoft Office - When Will Collaborative Tools Disrupt?

Last week, Microsoft announced that mobile and tablet versions of its Office applications would be free. The question is, should we care?

In the short term, the answer is yes.  Most companies and enterprises spend a good deal of money yearly on user computing devices and Microsoft licenses. Now let's say a number of those users are mobile, say the sales team. The new pricing enables IT to experiment and test Office with a small group to validate the experience on a tablet. If the mobile workforce can do enough of their document, presentation, and spreadsheet work on tablet devices then there is a strong likelihood that IT can outfit this group with tablets rather than laptops.

But the reality is that for the last several years, there have been less expensive alternatives to Office including offerings from Google, Zoho, Apple and now Amazon. For light weight users that don't need sophisticated features consumers and enterprises have had options if willing to change to a new user experience.

What about Power Users - What are their Needs Beyond Today's Office?

For power users, the question may be when, not if, there are suitable replacements to Microsoft Office or Office 365. Despite years of effort by Microsoft to enable collaboration and BI features in Office, most business users ignore these capabilities and end up working individually, passing documents back and forth via email, copying data into spreadsheets to do run off analytics and pasting charts into single use Powerpoint presentations. The aggregation of this this wasted digital effort will be the target for productivity improvements over the next several years.

The next generation of "Office" is already here, but the technologies go under different names and none have achieved critical mass as compared to MS Office. Until there are break away leaders that offer enough functionality to substitute MS Office and have a higher level of interoperability with other tools, the switching costs will look high to CIOs who can not afford a user backlash. Until then, these tools are largely additive to the Office experience. 

What is the next generation of "Office"?

It's collaboration tools that will aim to eliminate internal email dialogue and enable a more open, conversational, and searchable environment. Perhaps it will be a tool like Jive, an unseen mashup of Yammer, SharePoint, and Lync or a context enabled environment like Salesforce Chatter. 

Perhaps Excel 2013 will be more friend than foe in enabling spreadsheet jockeys to follow best practices in data governance and avoid a new generation of data landfills. My bet is on self service tools like Tableau and Qlik that realize that Excel isn't the enemy, it's PowerPoint, and both are investing in storytelling functions to eventually disrupt cutting and pasting into PowerPoint or other presentation tools.

Finally, perhaps either Google, Amazon, or Apple will shift from a Microsoft legacy orientation of documents, presentations, data and email to a more collaborative, mobile, and cloud enabled paradigm. Today, their apps look a lot like lesser versions of the Microsoft tools in order to chip away at Microsoft's dominance and easily switch over users. But if one of them re-imagined the user experience and reoriented their tools, it has a chance to get business users to think and act differently.

Place your bets.

continue reading "The Death of Microsoft Office - When Will Collaborative Tools Disrupt?"

Wednesday, November 05, 2014

Agile Data Scientist, A Disciplined Hiker or Reckless Hunter?

In my last post, I provided some guidance on why and how agile data practices can lead to better Business and IT collaboration. Agile aligns priorities, focuses multidisciplinary teams to hit goals, and enables teams to self organize and figure out individual responsibilities. As the team succeeds, management can then map our roles, responsibilities, and other governance considerations.

Data Insights are a Journey

As I ended my last post, I suggested that there are some fundamental differences between agile teams delivering a software driven product versus the analytics and insights that data science teams strive to deliver. Product teams are optimizing against scope, cost, and time to complete their delivery and agile teams often fix time and cost while making tactical decisions on scope. Unlike product and software deliveries, data teams are less likely to have structured deliverables like product launches or software releases. As I explained to CIOs contemplating their data strategy, data science and analytics is closer to a journey and not a destination or milestone. This key difference leads to a different way of thinking about agile data teams.

Hiking or Hunting Your Way to Insights

Picture a hiker in the wilderness who is trying to find the most interesting locations to photograph and is using her skills and tools to find vistas, waterfalls, and wildlife. When the hiker has a clean line of site to an interesting destination, she will move with vigor to capture it. Other times, she will navigate the dangers of the wilderness in a search, often stopping to check her gear, set up camp, or completing other necessities needed for a long term journey.

Data scientists are in the search for insights, and much like a nature photographer, know they've found something insightful when they see it. Until then they are on a search or hunt using a combination of their skills and data tools to support their discovery efforts. 

Now there are some very disciplined hikers who are well equipped and methodical in their approach. When faced with adversity, they have the tools and skills to address challenges without compounding to the risks they are facing. There are also more adventurous hikers that act more like reckless hunters; they are so fixed on the kill that will take on additional risk in order to meet their objectives. (Note: I should point out that there certainly are reckless hikers and many disciplined hunters out there. My point in the analogy is to illustrate differences in both behavior and persona.)  

Agile Data Scientists

The same is true for data scientists. Complexity lies in the form of slow data processing tools, technical difficulties in getting data integrated, structural issues with how the data is stored, data quality issues and other impediments that complicate data discovery efforts. Some data scientists will collaborate to improve the underlying tools, data structures, data processes, or other infrastructure barriers in order to achieve their current and future goals. Other more scrappy scientists focused on just getting the job done will engage in bad data practices, create silo databases or perform adhoc analytics.

So there in lies two major differences between product and data agile practices:
  • Product organizations march to milestones like launches and software releases, data science is more of a journey.
  • Agile product teams evolve products around a stable set of platforms and infrastructure. Data scientists have to choose if and when to be disciplined because there are many tools that can easily bypass defined data structures and practices.
These two differences are key to managing data science practices and big data technologies. More to come!
continue reading "Agile Data Scientist, A Disciplined Hiker or Reckless Hunter?"

Tuesday, October 28, 2014

The Agile Data Organization - Balancing Responsibilities in Data Science Programs

If you've read this blog or have seen me speak at a conference, then you know I am a strong proponent of self-service BI programs. I've posted on principles of self-service BI programs, attributes of data driven organizations, and how to avoid data landfills among many other big data topics all aimed to get business teams successful competing and driving decisions with data.

But success isn't driven by technologies, data practices, or the value of the underlying data alone. It is people and organizational structure that truly drive success and yet this is where I see many organizations make classic missteps. The problem is in balancing responsibilities and making decisions on who does what steps in the data management practices, and who owns what decisions.

Three classic mistakes

Here are some of the missteps some leaders and organizations make when considering how to manage big data or self-service BI programs:

  • An overreached business team that tries to cut out IT from all or the majority of data management practices. In other words, data scientists on business teams trying to to turn "self service" to "complete control"

  • An overgoverning IT team that tries to provide technologies and identify structured business practices on every step from data gathering, to processing to delivery.

  • An overzealous PMO that tries to identify and label every part of the process and formally assign responsibility and decision making before the practice is in place and business value determined.

Hopefully you can visualize what's happening here. If you elect to be the overzealous PMO, you have a lot of up front work to define structure, process, roles, and responsibilities. If you choose not to predefine a structured practice with roles and responsibilities defined, then the organization will evolve its practice through experimentation and attempts to provide value. This is generally a good "agile" evolution, however, it, can lead to an imbalance depending on who has more organizational power and controls. Undisciplined business teams with little IT participation can lead to the first scenario and an overly controlling, "technology first" approach yields the second scenario.

What's the Solution To Getting A Balanced Business and IT Data Organization?

First and foremost, organizations need to recognize that this is not a unique problem to self service BI or data science programs. Agile product and software delivery teams are almost always cross-functional between Business and IT. The heart of agile is the product owner managing a backlog of features and enhancements, defining minimally viable solutions, working with IT on implementation scenarios, and prioritizing planning and development stories. Strong agile teams also have mechanisms to express and prioritize technical debt, larger business investments, and more significant infrastructure changes. 

The same practice can be applied to Agile Data Organizations, except that instead of prioritizing features, organizations look to prioritize big data questions. What questions provide value to stakeholders and customers that are worth answering? How do we attribute value and estimate feasibility on answering the question? How do we factor in other work such as loading in new data sets, data cleansing efforts, or improvements in data processing?

The next step is to get a team working together on discovery efforts. Once a multidisciplinary group understands priorities, there is a stronger likelihood that they will work together and disregard organizational boundaries and responsibilities.

Want to get started? See my related post on agile leadership practices to help data scientists.

But that's the start. There are some fundamental differences between software and data analytics that also contribute to the organizational discord. More to come!

continue reading "The Agile Data Organization - Balancing Responsibilities in Data Science Programs"

Monday, October 20, 2014

Five Takeaways from Mobile Enterprise Boston

Last week I attended M|Enterprise Boston, a conference that brought together technologists ahead of the curve in mobile application development, IT executives looking for best practices on mobile device management in large enterprises, and leaders looking to help their business gain a competitive edge by developing differentiating mobile capabilities.

My takeaways -

  • Use Hackathons to get developer adoption - Mahesh Bala of Snyder Electric scheduled a hackathon to get developers across the globe to try out his mobile development platform. It's a brilliant idea for two reasons. First, for developers that were already developing mobile apps in his organization, the hackathon provided a venue to learn, tinker, and hopefully by into a standard development tool and methodology for developing future applications. For more novice developers, it was an opportunity to learn a new skill and gain confidence to develop mobile technologies. Given how hard it is to get a decentralized group of developers to adopt a development standard and how important it is to grow an enterprise's mobile development talent pool, this approach is simply brilliant. In addition to the innovation developed at the hackathon, I am certain Mahesh got value feedback from the developers on where to make platform improvements. 

  • Use Genius bars to help users - Apple's genius bars are a huge success in getting its users everything from basic support to important training and problem solving. Why not use the same approach in the enterprise? Brian Katz @BMKatz talked about his enterprise's approach to setting up internal genius bars to help its users fix issues and learn how to better use their mobile devices. Smart to be present and make it convenient for users to find technologists rather than wait or hope that they dial into Support.

  • Day in the life of a user - There was a good amount of discussion on the importance of understanding user needs and work flow before designing applications. The best and most simple advice I heard - and apologies for not remembering the source - was to have members of the team walk in the shoes of their target users and experience a day in their life. Why? Because for mobile applications to be useful and used, they have to provide significant benefits at the right time and place and a good way to understand their needs is to experience it directly.

  • Agile development of mobile applications was debated politely. Many participants stressed the importance of identifying user persona, needs, workflows and to design the user experience while others were more vocal on agile principles. The two are not mutually exclusive and all agreed that mobile app development should target a minimal viable product but needs to be good enough so that users don't download, disappoint and drop.

  • Mobile analytics is the key to understanding user behaviors and tuning mobile applications and possibly more important than web analytics. As Adrian Bowles @AJBowles put it so well, the intersection of mobile and analytics is being "aware" so that the app is always on and collecting data and "everywhere" the user goes you have the ability to provide value and capture insights. The combination of mobile with analytics, assuming strong privacy considerations is a strategic differentiating tool. 

continue reading "Five Takeaways from Mobile Enterprise Boston"

Monday, October 06, 2014

The Future of Works Starts with Fixing Bad Meetings

How do you manage meetings so that they are productive and complete with documented decisions and prioritized follow up tasks?

This is a big, but an important topic. A lot of people's time and a corporation's spend is burned in meetings, yet most people can recount the bad meetings that were a waste of time versus others that had tangible outcomes. 

Why is it that executives with years of experience and advanced degrees struggle with the simple task of gathering the right number and mix of people, establishing an agenda, managing the meeting time, and insuring the proper documentation is disseminated?

I can hear some people saying that agenda-less meetings are also important. Blue sky meetings? Catch up sessions after a long break? Agile standup meetings that have a defined protocol? I agree that there are many meetings that may not require formal definition, but still require some structure. 

Is the Problem People or Technology?

Today, there is relatively low cost to set up a meeting. Jump on Microsoft Outlook, identify participants, find room on their schedule, book a meeting room and add a one line Subject. Most of the work is tied to answering who, when and where - very little on the what! Microsoft wanted to make it easy on business users to schedule meetings so the focus is on logistics and very little substance is required.

So what if we used technology and implemented some rules?

  • Every meeting requires an agenda - either a description or ideally a schedule of topics
  • Scheduling a meeting requires identifying two roles - A leader, for the person in charge of the meetings agenda and outcomes, and a Scribe, the person required to document decisions and follow up tasks.
  • Meeting costs calculated - Most meetings have too many people invited. I'd prefer implementing some form of governance, like meetings can't have more than eight people without an approval, but this may not be practical. Instead, what if the cost of the meeting was transparent to its leader? Even if the cost is calculated off of a simple flat hourly rate for all employees, publishing the cost will give the leader a sense of accountability.
  • Meetings require check in - when they arrive at a meeting. Better yet would be to install beacons in conference rooms to automatically record check ins. Dialing in or using a virtual meeting room? These tools could be configured to automatically log in the check ins.
  • Meeting outcomes are documented - and collected by the Scribe in a centralized tool that captures decisions and assigns follow up tasks. By definition, the Scribe is automatically scheduled time after the meeting to complete this documentation. On completion, participants are automatically emailed a link to the finalized meeting report that includes its agenda, participants, decisions, and follow-ups.

But It's People...

While this technology helps, it's peoples behaviors that also need to change. Some of these tools are expensive to implement (beacons) and can easily be circumvented. So this will require collective participation, collaboration, and a little bit of self policing to make a transformation successful.
continue reading "The Future of Works Starts with Fixing Bad Meetings"

Monday, September 29, 2014

What Technologies Work Best for Decentralized Data Scientists?

If data scientists, analysts, quants, or BI specialists are in a centralized department, then that group can staff and train its members to support one or more technologies based on business need. Technologies such as data processing, analytics, statistics, visualization, or data mining are good examples. 

But what happens when these resources are scattered across multiple departments. One department may have an expert data scientist, another may have a small group doing internal reporting, and a third group might have outsourced its analytic function. If data scientists in the organization are decentralized with different goals, skills, and operating models, can IT still provide a common set of Big Data and analytic tools and services to the organization and support these different functions?

The answer is yes, but decentralization leads to a different set of technologies and IT services. Since different users will have different goals, capability needs and skills, IT needs a Swiss army knife of data management and analytic technologies and related services

Self Service BI - The Analytic Swiss Army Knife?

That Swiss army knife has come with new technologies branded as "self service" BI that aim to enable business users - and not IT - to solve many data processing, analytics, or visualization tasks. The software companies developing these technologies recognize that IT can be a bottleneck to solving data challenges and have developed products that take the coding out of data tasks. With these tools, you can aggregate data sets, perform joins, cleanse data, map data, perform analytical calculations, identify trends, seek outliers, and develop dashboards - all with minimal coding!

Data scientists working in different departments can make great use of these tools. Imagine one in marketing that can blend their marketing database with a social networking feed to develop insights on prospects? Consider someone in sales ops who develops dashboards for sales directors making it easier to understand and action the sales pipeline? A financial analyst can develop common reporting dashboards and departmental specific reports.

But these tools deployed without defined practices and governance will create a new generation of potential data silos, bury analytical calculations, create another form of spreadsheet jockey, or produce too many dashboards. They will create work-arounds to performance issues or  duplicate data in order to make today's analysis more convenient. They might expose sensitive data to too many people in the organization or violate privacy or compliance constraints when moving or storing data.

The role of IT in Self Service BI Programs

So with these great tools comes even greater responsibilities. For brave technologists and CIOs embracing a decentralized data strategy, the task does not end with identifying talent, selecting and implementing "self service" technologies, and training. It must define new data practices and governance, clearly identifying the responsibilities of business users and demonstrating the value of IT by providing a matching set of data services.

Where are workbooks versioned? How are analytical calculations published, validated, and tested? How does one request assistance integrating new data or help solving a query performance issue? How are new tools evaluated and upgrades tested? What types of documentation is required, where is it published, and how often is it updated? How is security enabled? How does the organization measure data quality? What visualization standards will make it easier for enterprise users to leverage data and dashboards in their decisions making?

These questions need technology solutions and service definitions. The CIO needs to define a new set of data management practices and lead the organization to be more data driven.

I suspect that as organizations become more data driven, the more data science skills will be needed, the more likely they will be deployed across the organization and therefore more likely self-service BI programs will be established.

continue reading "What Technologies Work Best for Decentralized Data Scientists?"