BIG DATA – Analysis of huge amounts of data for detection of threats and decision making

This post is also available in: עברית (Hebrew)

National intelligence services, acting in the service of homeland defense, collect information from a huge variety of collection sources: sensors, video recordings, listening-in, manned and unmanned aircraft, border observations etc. A huge amount of information is stored in databases and “data warehouses” of the organization.

It can be assumed that 90 percents of the information are practically worthless when there is silence over the border and “everything is OK”. But one defending his borders looks for exceptional activity,  that may indicate intentions for attack, planning of terror acts or initiation of a cyber attack. That means: the same intelligence service needs means for collection of data, its storage, filtering, and fast access to the information in order to analyze it for detecting exceptions, as fast as possible. An exception can indicate an evil intention.

Here enter the BIG DATA technologies, planned to take care of the extensive amounts of data (measured in Peta-bytes and more), and this is the subject of  the conference on BIG DATA, organized by ORACLE and I-HLS, with the participation of  select companies dealing with the subject in Israel.

i-HLS ISRAEL Homeland Security 

Avner Maimon, business development manager of ORACLE Israel, explains that the tendency today is to understand trends and identify intentions and threats in the network before they exist, and for this purpose it is required to collect a lot of information from various sources, including blogs, security cameras, sensors, emails, information flowing in the organizational network, cellular calls, etc. By analysis of all this abundant material it is possible, by means of BIG DATA technology, to identify and detect trends and potential cyber threats. Hostile activity can also take place in social networks. When suspect traffic is suddenly detected in the network, a group of people become organized in a certain area, somebody is using a password many times trying to penetrate information files – these are indications of impure intentions, and the BIG DATA technologies enable to analyze exceptions and send alerts. So also in the field of cyber war: a lot of material is going around in the network, and you have to pick the essential information from the lots of garbage and unnecessary information contained in the network. However, the amount of garbage and unnecessary information is enormous and therefore the job of filtering can be exhausting up to impossible without BIG DATA technologies.

There is distinction between two types of data:

• STRUCTURED DATA – data that has a structure and is stored in traditional data bases known for many years.
• UNSTRUCTURED DATA – data that is not structured, collected in social networks, data files, blogs, WORD documents, and any other data that is not managed in regular data warehouses.

Collection- management – analysis

Avner Maimon: “The idea today, and this is the task of companies like ORACLE, is to unite the two kinds of data in order to provide the customer a unified and comprehensive view of all the information disposed by the client organization. The solution of ORACLE is unique in the sense that it gives the customer a complete solution combining data collection, data management and execution of analysis on the (structured and unstructured) data, to enable the customer to make his decisions. We are dealing here with a collection of hardware and software products, providing a full end to end solution. The organization submits its requirements, and ORACLE will tailor a BIG DATA package suited to his needs, as a complete solution to BIG DATA challenges. And the challenge is coping with three components of BIG DATA: Volume, Speed and Variety. It has to do here with a lot of diversified information (pictures, log files, XML, tables), to which the customer will require very fast access for getting the information he needs. To this end solutions are required that will enable him to store, manage and analyze with low operational cost and maximum of performance.

i-HLS ISRAEL Homeland Security 

ORACLE has developed special tools like ORACLE BIG DATA APPLIANCE CONNECTORS for storage, management, moving and analysis of unstructured data based on ORACLE developed software on top of CLAUDERA HADOOP technologies, ORACLE ENDECA for searching, identification of new trends and execution of analyses on the filtered information, at top speed. The system is used currently by about 25 customers in Israel.

This field is part of new branch of science – DATA SCIENCES, looking for new trends to identify characteristics of the data in data warehouses, using the innovative graphical capabilities with tool s like ORACLE ENDEA and ORACLE R, which facilitate the identification of special characteristics, so that by advanced visual means it is possible to see the trends and to take decisions based on these analyses.

By: iHLS NewsDesk