Version 9 of Sinequa ES Now Available
Hadoop and Mahout integration, smart caching, Semantic Markers, and "hyper-indexing" lead to "elastic" high performance, high relevance and ease of use in big data search & analytics.
Paris, 25 June 2014 - Sinequa, a leading real-time big data search & analytics software company, announced today that they have released Sinequa ES Version 9, a major new version of its solution software.
Sinequa ES Version 9 offers a whole range of new functionalities, above all the integration and use of Hadoop, as well as many features inspired by the most innovative ideas from recent customer projects. The Sinequa R&D team has continued to develop new connectors to data sources, now at a total of 140 (including PTC Windchill, Mongo DB, Scality, Office 365, box...), to refine language analysis in the 19 languages covered by Sinequa, specifically in Asian languages including Chinese, Japanese and Korean. Furthermore, Version 9 offers advanced geo-location functions, tying products or people to locations and taking distance into account in relevance rankings. It also integrates with Amazon Web Services (AWS) such that customers can benefit from an "elastic" Sinequa grid hosted on the Amazon cloud and from certain services specific to AWS. The "elasticity" of Sinequa on AWS lets customers instantly scale computing resources to their requirements at each moment in time, be it for the indexing a large new data source or when adding a large number of users.
"The release of Sinequa ES Version 9 is a major step forward, in particular through the Hadoop / Mahout integration, that re-enforces our position in the big data arena by offering automatic classification and clustering, recommendations, and predictive analysis," explains Alexandre Bilger, CEO of Sinequa. "Our work on Hadoop does not signal any change concerning our positioning in real-time search & analytics. Our users expect and demand real-time service. With the new version we have further enriched our index, even created a "hyper index" by indexing our index, in order to provide users with more concentrated information in real time, while simplifying user interfaces for end users and administrators alike."
Sinequa ES V9 offers Hadoop integration on three different levels:
The Hadoop File System HDFS can be accessed as a data source via a new Sinequa connector.
Bi-directional Hadoop integration: Sinequa ES V9 can index data from Hadoop, but additionally, the Sinequa index can also be accessed by Hadoop for "typical" Hadoop processing: The calculation of relevance rankings and recommendations, and for predictive analysis. Moreover, Sinequa ES V9 can use Hadoop calculations for "smart linguistic indexing" and "index re-composition" according to clever but compute-intensive algorithms making use of company- and trade-specific knowledge (dictionaries, ontologies, taxonomies, directories).
With Hadoop Mahout (machine learning) we can leverage algorithms for automatic classification, recommendations, and predictive analysis.
For automatic classification, users can provide a large corpus of already classified documents to Sinequa/Mahout and ask the system to classify new incoming documents "the same way". If the system gets it wrong, users make corrections, and the system will refine its classification by taking these corrections into account. This method of machine learning in classification is helpful when large amounts of classified/categorized documents already exist, while users find it difficult to express rules for this classification. These difficulties may come from different views on document sources by different professional communities. The existing classification serves as a "de facto classification method".
A number of performance optimizations have been introduced in Sinequa ES V9, some of them the result of work in the most innovative customer projects where they have proved highly effective and useful.
Amongst these are Intelligent Caching mechanisms and hyper-indexing. They help Sinequa ES V9 overcome two serious challenges to real-time responses:
Some data sources have not been designed for fast data extraction on a massive scale. Sinequa ES V9 introduces a smart caching mechanism for data from such sources. This combines the advantages of search and "elastic" storage. Re-indexing with new analytic concepts is no longer hampered by slow data sources and offers "persistent insight". Rules can be defined for the data to be extracted and to be refreshed in the cache.
Extracting relevant information for knowledge workers, such as scientists in the pharmaceutical industry, requires knowledge of synonyms and related topics in areas as diverse as diseases, genes, drugs, molecules, mechanisms of action, etc. Using a "shot gun approach" for finding such relevant information by simply launching as many queries as there are synonyms and related topics would never produce results at the required speed. That is why Sinequa builds a "semantically rich index" that uses company and trade knowledge of a subject domain in order to aggregate information on synonyms and related concepts.
In addition, creating a "hyper index" by indexing the "original" index, allows storing the complete "fingerprint" of a person extracted from a large corpus of documents. This fingerprint contains the areas of expertise of a person and the topics he or she has worked on over time. Consider it a "semantic join" of persons and topics that appear together in documents. The documents themselves are no longer part of the join, making the hyper-index very compact and its retrieval extremely fast. This allows a simple query (centered on just one topic) to deliver information on all semantically related topics, and on the best available experts on a given topic in just about one second, even when dealing with big data.
A pharmaceutical company like AstraZeneca can get a real-time view of the best team of experts on a research topic and related subjects (e.g. disease, genes, drugs, active molecules, mechanisms of action, lab tests, clinical tests, etc.) Users can even tweak the relevance of related subjects by moving sliders on their screens and see the "dream team" change in real time.
For more information on Sinequa ES V9, contact sinequa-es-v9(at)sinequa.com