This post is also available in: heעברית (Hebrew)

The proliferation of real-time data from a wealth of sources such as mobile devices, web, social media, sensors, log files and transactional applications, has brought about the development of advanced Big Data technologies for various military mission-critical applications.

Big Data technologies include distributed computational systems, distributed file systems, massively parallel-processing (MPP) systems, and data mining based on grid computing, etc.

The big data techniques used by businesses to mine databases are being adopted by the US military to glean more information from many different types of data sources—from drones, automated cybersecurity systems, terrorist databases and many other sources. The technology will not only help warfighters on the battlefield, it will also be used to improve fields ranging from software development to vehicle maintenance.

According to, the US Department of Defense announced a FY 2017 budget request to invest in “Big Mechanism,” an effort to automate computational intelligence for biology, cyber, economics, social science, and military intelligence.

As per the budget, Defense Advanced Research Projects Agency (DARPA) projects that its planned investment related to big data will rise from approximately $216M in FY 2015 to nearly $269M in FY 2017, a planned increase of 24.5%.

In 2014, DARPA announced a program it calls Mining and Understanding Software Enclaves (MUSE) to improve the quality of the military’s software. It wants to develop “big code” software packages capable of managing information without distorting it or collapsing under

the weight of the large data sets, the so-called big data, which is at the heart of the Obama administration’s technology programs announced in 2012.

Various Pentagon big data initiatives include research by the Defense Threat Reduction Agency to track dangerous pathogens, a DARPA-directed plan to mine health data to track cancer cells, a combination of data mining and patent analysis to track “disruptive technologies” that could alter the future of military equipment and planning and counter-force programs to track the spread of weapons of mass destruction by DTRA.

Some Big Data Tools & Technologies:

  1. Apache Object Oriented Data Technology (OODT) for non-programmers to create, edit, manage and provision workflow and task execution.
  2. Apache Bahir:  Serves as a home for existing connectors that initiated under Apache Spark, and provides additional extensions/plugins for other related distributed system, storage, and query execution systems.
  3. Conditioned-based Maintenance (CBM): The US Air Force officials, data scientists, and other personnel are harnessing advanced data-handling and analytics tools, both software and hardware solutions, to streamline workflows, increase efficiencies and productivity, track and manage assets, and replace scheduled/time-based maintenance with conditioned-based maintenance (CBM) – saving at $1.5 million in one year. Specifically, Air Force officials opted to use Teradata’s Aster platform to better manage Flightline maintenance; Inventory; and Aircraft depot data.
  4. Hadoop software.
  5. Organizational, Relationship and Contact Analyzer (ORCA): Developed by U.S. Military Academy, this software was initially developed and used in military operations to identify networks of insurgents, and is now being used domestically as a way to hone the software for future wartime applications, while in the meantime providing police gang units with a valuable free tool that could eventually see a more widespread deployment.
  6. Tableau: A very effective tool to create interactive data visualizations very quickly, to be used by developers as well as non-developers.
  7. D3.js: A JavaScript library that is used for data visualization.
  8. Power BI: Power BI allows developers to create visualizations and display data in a very accurate way that you will not find such facilities in other BI tools.
  9. R language: A language and an environment to run statistical calculations and produce data graphics.