Tune the solution at this stage if the solution you

 Tune the solution . At this stage, if the solution you have created is working correctly and the
results are valuable, you should decide whether you will repeat it in the future; perhaps with
new data you collect over time. If so, you should tune the solution by reviewing the log files it
creates, the processing techniques you use, and the implementation of the queries to ensure
that they are executing in the most efficient way. It’s possible to fine tune big data solutions to
improve performance, reduce network load, and minimize the processing time by adjusting
some parameters of the query and the execution platform, or by compressing the data that is
transferred over the network.
 Visualize and analyze the results . Once you are satisfied that the solution is working correctly
and efficiently, you can plan and implement the analysis and visualization approach you require.
This may be loading the data directly into an application such as Microsoft Excel, or exporting it

Planning a big data solution 33
into a database or enterprise BI system for further analysis, reporting, charting, and more. For
more details, see Consuming and visualizing data from HDInsight .
 Automate and manage the solution . At this point it will be clear if the solution should become
part of your organization’s business management infrastructure, complementing the other
sources of information that you use to plan and monitor business performance and strategy. If
this is the case, you should consider how you might automate and manage some or all of the
solution to provide predictable behavior, and perhaps so that it is executed on a schedule. For
more details, see Building end-to-end solutions using HDInsight .
Note that, in many ways, data analysis is an iterative process; and you should take this approach when
building a big data batch processing solution. In particular, given the large volumes of data and
correspondingly long processing times typically involved in big data analysis, it can be useful to start by
implementing a proof of concept iteration in which a small subset of the source data is used to validate
the processing steps and results before proceeding with a full analysis. This enables you to test your big
data processing design on a small cluster, or even on a single-node on-premises cluster, before scaling
out to accommodate production level data volumes.
It’s easy to run queries that extract data, but it’s vitally important that you make ever y effort to validate
the results before using them as the basis for business decisions. If possible you should try to cross
reference the results with other sources of similar information.
Is big data the right solution?
The first step in evaluating and im plementing any business policy, whether it’s related to computer
hardware, software, replacement office furniture, or the contract for cleaning the windows, is to
determine the results that you hope to achieve. Deciding whether to adopt a Hadoop-based big data
batch processing approach is no different.