Team Data Science

The Fix Is In

photo: JD Hancock

Big data are a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization.

So say the opening sentences of the “Big data” article in Wikipedia. The people primarily responsible for conquering those challenges are data scientists. Being a data scientist these days is rather like being a Renaissance person; one must possess knowledge of and competency in a wide variety of subjects related directly and indirectly to the fields of mathematics and science.

Fortunately, data science has a number of sub-specialties to share the load. Understanding–defining–who does what (capturing, curating, storing, searching, sharing, transferring, analyzing, visualizing, processing) and why they do it means companies building data science teams can intelligently choose the areas of specialization that will best serve their goals.

Five Roles You Need on Your Big Data Team

Of course there’s the data scientist, the coveted knight in shining armor who visualizes models and creates (and continuously optimizes) sophisticated algorithms to transform data into something useful. But she could not do her part to fulfill corporate expectations without the support of equally coveted

  • Data hygienists, who deal with the “dirty data” problems inherent in collecting data so the data is clean now and stays clean in future.
  • Data explorers, who burrow into all the data a company collects to determine what, if anything, can be done with it, including how data originally collected for a different reason might be repurposed.
  • Business solution architects, who structure and organize data so it’s properly updated and where it needs to be within the necessary timeframe of every query–a critical feature of today’s data science when queries are ‘answered’ in real-time.
  • Campaign experts, who, with an in-depth understanding of both the technology and marketing, can turn the knowledge derived from data into insight and then into advice.

Assembling a powerful data science team, whether that team is internal or third-party, is necessary to applying big data tools. However, success rests as heavily in the hands of the right corporate culture as it does in the right specialized people. The best solution? Welcome reevaluation, innovation, experimentation, and keep the focus on the end game.

Scarce and growing scarcer

Being able to use the knowledge derived from data, achieving the insights to which data can lead is the centerpiece of a marketer’s requirements in the big data era. But before you can acquire knowledge, you must understand the data itself and how its patterns fit together and suggest other patterns, how to work with it to produce useful, meaningful knowledge. Enter data scientists.

Employment opportunities for data scientists are growing. They will continue to grow, and some institutions are putting educational programs in place to help meet future demand. However, projections suggestion the demand for data scientists will soon exceed their availability. A compelling graphic synthesizes the problem.

Top skill set Requirements to be a Data Scientist

Data scientists aren’t data analysts. While the two roles may start with a grounding in scientific and mathematical skills, a data scientist is far more a “Renaissance individual who really wants to learn and bring change to an organization,” says Anjul Bhambhri of IMB. About a data scientist’s skill set, Mark van Rigmenam writes,

They need to have statistical, mathematical, predictive modelling as well as business strategy skills to build the algorithms necessary to ask the right questions and find the right answers. They also need to be able to communicate their findings, orally and visually. They need to understand how the products are developed and even more important, as big data touches the privacy of consumers, they need to have a set of ethical responsibilities.

Often, related fields of study pair with a breadth of programming, managing, processing and curating skills to shape the qualities of individuals who will guide a business’s effective use of data. Rigmenam suggests an ideal data scientist would have the following skills.

  • Strong written and verbal communication skills;
  • Being able to work in a fast-paced multidisciplinary environment as in a competitive landscape new data keeps flowing in rapidly and the world is constantly changing;
  • Having the ability to query databases and perform statistical analysis;
  • Being able to develop or program databases;
  • Being able to advice senior management in clear language about the implications of their work for the organisation;
  • Having an, at least basic, understanding of how a business and strategy works;
  • Being able to create examples, prototypes, demonstrations to help management better understand the work;
  • Having a good understanding of design and architecture principles;

We would add, while an effective data scientist requires latitude to consider and experiment (work autonomously), she must also be able to work cooperatively. Data scientists are members of teams that aren’t simply made up of senior leaders. There are plenty of other employees who work in the trenches with ideas about situations that require solutions and how solutions would fit into goals of other departments. Failures in cooperation and communication can lead to costly disasters.

Likely, few data scientists possess all the above qualities, so a business should prioritize the ones important to them.

In planning for apply new technologies, businesses must also plan for how they will apportion responsibilities for critical data science needs–through third-party applications or data-science-specific internal departments or perhaps, both. At present, we are gazing at the tip of the big-data, data-scientist iceberg. Demand for big data solutions is increasing. So is the demand for the innovators behind the solutions.

Let’s discuss: The closer you think you are, the less you’ll actually see

1968 Dodge Charger R/T - 2560x1600 Desktop Wallpaper Black Background What sort of search engine magic does a company like Google work to deliver results meant to please and satisfy those who use its services? Truth is, it isn’t magic at all, and neither are the finer points of accomplishing this perceived magic very magical.

Google’s Search Magic Revealed

Details obfuscate purpose, and correlations can often obfuscate the bigger picture: causation.

When you try to determine how best to entice and engage customers, the search engine optimization tactics you employ have a way of distracting you from what is most important. So many correlations, so many microscopic examinations of efficacy, so much attention to a surfeit of arcane SEO advice can lead to a myopic view of how customers want to interact with you. Like a magician’s slight of hand, these tactics redirect your own attention from the true magic that is happening elsewhere. In essence, the closer you think you are, the less you’ll actually see.

When a business focuses on an over-abundance of ‘expert’-recommended small stuff, it typically misses the point, which is: How can you provide the best product for your customers? How can you satisfy, even delight, your customers? How are you going to make each and every customer feel you are there just for them?

The goal of your efforts shouldn’t redirect the attention of your customers. It should be based on moving beyond mystification to deliver the experiences your customers actually want and tailor those experiences to each customer, one at a time. ‘Big data,’ that merger of lots of data points with a variety of data sources and real-time delivery, is your best opportunity to move beyond correlation and into the big-picture realm of causation: If you understand more of what your customers want, you are in a better position to deliver it. The secret both you and your customers are looking for, the secret a company like Google has long since discovered, is customer-centricity.

Is slight of hand the answer? Hardly. Once you understand the magic behind the magic, you learn how to identify the reasons why your customers should care about you and buy from you in the first place. Then you don’t have to deliver the magic of magic; you can deliver the magic of value.

Image: by Barry Wetcher, SMPSP – © 2013 Summit Entertainment, LLC. All rights reserved

10 Qualities a Data-Friendly Business Culture Needs

[ New Perspective ] Tokyo Metropolitan Government Building, Shinjuku, Tokyo, JapanUsing and continuously optimizing data with the long-term goal of providing customer value and increasing conversion sounds easy; it can be difficult to execute. Getting the most from your data requires hard work and a willingness to adapt, to experiment, to learn from mistakes, to correct those mistakes as quickly as possible and to keep doing the process over and over without end. But, no matter the scale, crunching data to generate information and insight is valuable only if the organizational structures are in place to support the effort.

One of the most important problems corporations face is creating, at all levels of leadership a framework of support for and participation in new technologies that can help refine and expand how they do business. As pressure to adopt big data technologies grows stronger, many organizations have understandable fears concerning the integrity of their businesses. At the very least, leadership often perceives these technologies as threats to institutional knowledge and continuity.

Alex Miller of QVC, Anthony Bucci of RevZilla and Slava Sambu of Office Max have offered insight into the benefits of staying current with data processing technologies. They have discussed some of the stumbling blocks they have faced. But, as these three suggest, stumbling blocks are not dead ends. Organizations able to confront problems toward finding solutions soon begin to reap the rewards associated with using data effectively.

One of the goals—and it will become increasingly important—is corporate agility. With the ability to analyze and operate in real-time, a business can be far more responsive to its customers. It can quickly evaluate what is or is not working and correct the problem. It can identify problems and fix them immediately. For example, some data a business collects is going to be “dirty.” This is, and has always been, a common problem. Implementation of information derived from problematic data can potentially create a large problem with a large impact on the bottom line. A company needs to be agile enough to catch these problems, evaluate their nature and devise appropriate solutions as quickly as possible. As in right then; better still, an hour ago. Waiting while information makes its way slowly up the chain as each management level makes a decision—even waiting as long as it takes to organize a meeting of department heads—is not an option anymore.

Businesses are accomplishing this every day, and some have been employing these practices for years, decades. Amazon operates like this. Online newspapers operate like this. Alex Miller’s discussion of the need for immediacy in real-time data management of live broadcasting is particularly relevant. Real-time responsiveness is possible and advantageous with a sympathetic business culture.

What makes for a corporate culture that can successfully welcome and accommodate the emerging landscape of using data?

  1. An ability to understand a data-driven focus can out-perform many, if not all, previous business solutions
  2. An open-mindedness that supports the research, development and experimentation necessary to make best use of big data
  3. A perspective that supports the idea operating in real-time is an excellent way not only to enhance the customer experience but also monitor for problems and quickly correct them. However difficult it can be to negotiate at all levels, business agility is critical
  4. A marketing optimization framework that allows marketers to use the data effectively to make marketing decisions in real-time. Companies with higher conversion rates almost always have better marketing efficiency ratios (net contribution/marketing expenses). These companies understand it’s hard work to accomplish better marketing efficiency ratios, but it’s considerably more lucrative to do so.
  5. Higher standards of accountability throughout the organization, up to and including the CEO. Does the CEO know which factors of the customer experience impact sales, which projects or departments to favor, what truly needs to be done to optimize the marketing efficiency ratio? In a data-driven business climate, the CEO must know these things
  6. Leadership that promotes higher levels of communication, even collaboration, across all teams
  7. A willingness to grant certain decision making powers to smaller teams
  8. A commitment to employing conversion rate marketing principles
  9. An emphasis on making sure data as well as information and insight derived through analytics are flowing to individual teams across the organization so each can make clear decisions and execute in real- or near-real-time
  10. People and processes that foster a culture of risk-taking and ongoing testing

It took very little time to list (or read) these qualities. It will probably take much more time to internalize them so they become ingrained business practice. If you hope to stay competitive, however, just don’t let it take too long.

Blowing a Lot of Hot Air Over Big Data

Break freeShopycat. A feature for scanning the social media preferences of a customer’s Facebook friends and suggesting gift ideas sold on Walmart.com’s website. While Walmart has never shied from using analytic technology, it sort of missed the use-the-data digital boat when it came to focusing primarily on the customer experience. It isn’t alone. Difficultly integrating channels is an omni-channel challenge for many businesses with a brick-and-mortar arm. But Walmart is planning to make the shift to a data-driven organization (in fits and starts; selling certain items only online is not a great idea.) Walmart is hoping eighty-seven newly hired “engineers and coders” are going to help turn around the game it’s currently losing playing catch-up with (don’t lose more customers to) customer-centric Amazon.

Why Walmart Is Worried About Amazon

Let’s automate data prep; let’s ditch the internal chain of command that strangles experimenting with new ways to gather and use new data sources; let’s look at AI technology so resulting information and insight is built into the reporting process. That creative, forward-thinking analysts wish for intelligent tools to benefit a company is a clue they’re not just data jockeys and might be deserving of some attention. Big data tools now exist, or are in development, to answer most of the wishes. One thing is missing. Guess what.

Analytics Wishlist: Five Tools, Capabilites Analysts Wish Existed

Big data technologies are helping humans identify their own health troubles and find solutions. It makes sense that those with beloved barking companions wish for a similar technology geared to their four-footed buds. Now, your dog can wear a little ninety-nine buck device called Whistle. Whistle monitors your pet’s activity and compares the data to a large pool of other data to help you spot problems before they become serious. Score one for big data. (Warning: seriously adorable puppy alert)

Whistle Uses Big Data to Help Keep Your Dog Healthy

Is Verizon your telecom provider? If yes, did you know that since April 25 and until July 19, under a court order based on the “so-called ‘business records’ provision of the Patriot Act, 50 USC section 1861,” the NSA is collecting lots of personal datafrom your phone usage. Some folks mind the kind and quantity of personal data they are handing over, sometimes without even knowing they are doing it. Others accept that privacy in the big data world is becoming extinct. That’s a good thing, right? Up until it’s a bad thing.

Who’s Watching You? Not Just the NSA.

And Now for Something Completely Different

The art of air in motion: a real-time data visualization.

Wind Map

photo by: aussiegall