More than 90% of all data in an enterprise is unstructured information. This encompasses telephone conversations, voicemails, emails, electronic documents, paper documents, images, web pages, video and hundreds of other formats. Unfortunately, attempts to leverage this immense and strategic resource often fail because many businesses lack the requisite technology to understand and effectively utilize content that resides outside the scope of structured databases.
Similarly, unstructured processes are equally unwieldy yet comprise the bulk of business operations. Current trends anticipate the rapid proliferation of rich media, widespread adoption of VOIP, growing use of IPTV and increased scrutiny of white-collar crimes. This overwhelming growth demands an automated solution that can effectively manage an unstructured digital morass.
These concerns necessitate an information infrastructure platform that addresses all classes of information in a manner analogous to well established methods for structured databases. Akin to the Relational Database Management System (RDBMS) that revolutionized the computing industry in the 1960s, this innovative platform enabled computers to process not only structured data, but also vast amounts of semi-structured and unstructured information using a global relational index.
Autonomy's ability to process all forms of digital information on a single platform offers a unique solution to a growing number of applications and devices that are increasingly dependent on utilizing unstructured information. Autonomy employs a unique combination of technologies to enable computers to form a contextual understanding of all digital content, as well as understand people's interaction with the data. Autonomy's technology eliminates the traditionally manual and costly operation of processing and analyzing information by performing these functions automatically and in real-time. This represents substantial savings for every type of organization and industry and is driving the accelerated adoption of Autonomy's technology across a diverse range of vertical markets.
"Autonomy does an exceptional job at analyzing unstructured data"
Technology Review, 2011
SEARCH : CLUSTER : CATEGORIZE : LINK : ALERT : PROCESS
Autonomy maintains an open philosophy with regards to the techniques it uses and is dedicated to selecting methods which optimize its technology, whether they are old or new. Autonomy embraces traditional or legacy methods such as keyword, Boolean, parametric and others. However, Autonomy is best known for its pioneering work in conceptual search based on computational pattern recognition (non-linear adaptive digital signal processing) and contextual linguistic analysis.
Built upon the seminal mathematical works of Thomas Bayes and Claude Shannon, and on a range of innovations that are covered by 170 patents, Autonomy technology identifies the patterns that naturally occur in text, voice and video files based on the usage and frequency of terms that correspond to specific concepts. By studying the preponderance of one pattern over another, Autonomy's technology understands that there is X% probability that the content in question deals with a specific subject. In this way, Autonomy extracts the content's digital essence, encodes the unique "signature" of the concepts, and enables a host of operations to be automatically performed on emails, phone conversations, video, documents and even people's interests.
Thomas Bayes was an 18th century English cleric whose work has become a central tenet of modern statistical probability modeling. Bayes' efforts centered on calculating the probabilistic relationships between multiple variables and determining the extent to which these relationships are affected when new information is obtained.
A traditional statistical argument posits that if a coin is tossed 100 times and comes up heads every time, it still has an even chance of coming up tails on the next throw. An alternative, Bayesian approach, is to say that 100 consecutive heads are evidence that the coin is biased. What Bayes' theorem clearly demonstrated is that: a) the more information given, the more accurate the view of the world will be, and b) prior experience should be used to inform new data.
In a typical problem such as judging the relevance of content to a given query, Bayesian theory dictates that this calculation be related to details that are already known.
A good example of this theory at work is Autonomy's agent profile technology. Users can create agents to automatically track the latest information related to their interests, and IDOL determines the relevance of a document based on the model of the agent.
Adaptive Probabilistic Concept Modeling (APCM) algorithms are also used to analyze, sort and cross-reference unstructured information. In a similar manner, knowledge about the documents deemed relevant by a user to an agent's profile can be used in judging the relevance of future documents.
While most other models start with a prior knowledge of the state of the system and apply training to it, Autonomy begins with a blank slate and allows incoming data to dictate the model. In true Bayesian fashion, the model mixes new information with a growing body of older content to refine and retrain the engine.
Shannon's Information Theory forms the mathematical foundation for all digital communications systems. Claude Shannon stated that information could be treated as a quantifiable value in communications. Natural languages contain a high degree of redundancy or nonessential content. For example, a conversation in a noisy room can be understood even when some of the words cannot be heard, and the essence of a news article can be grasped simply by skimming over the text. Information Theory provides a framework for extracting the concepts from this redundancy.
Autonomy's approach to concept modeling relies on Shannon's theory that the less frequently a unit of communication occurs, the more information it conveys. Therefore, ideas, which are rarer within the context of a communication, tend to be more indicative of its meaning. It is this theory that enables Autonomy's software to determine the most important, or informative, concepts within a document.
Performance of IDOL's Conceptual Retrieval
Built on a unique pattern-matching technology, IDOL's conceptual query mechanism allows a seemingly simple query expression to be evaluated in complex ways; as well as the matching of the basic terms within documents using patented weighting algorithms, it is able to develop the terms to 'read between the lines' and determine conceptual matches that legacy search engines would be unable to locate.
However, IDOL is able to perform these evaluations with surprisingly little overhead above the equivalent keyword query. The reasons for this are two-fold. Firstly, the majority of the work in the calculation and initialization of the conceptual matching is done at index time, as opposed to query time; the documents are analyzed while the data is being processed to form a statistical 'pool' from which queries can draw key conceptual information, as well as an overlying Bayesian network in which apparently unrelated pieces of information are automatically linked via dynamic probabilities. The second reason is that the documentmatching algorithm itself within IDOL uses widespread "short-circuiting" and iterative calculation to ensure that it only performs exactly as much calculation as is required. In essence, the key conceptual information is already available before the query has even started, and once it does begin, it feeds directly from the statistical core to load the information. The uniqueness of the query then forces the only truly complex step, a one-off calculation in which combination algorithms arrive at the most relevant set of documents to the query. These can then be returned without the need for looping through every potential match.
Some vendors only offer "black box" solutions, mistakenly believing that their technology can always provide the best answers with no tuning required. However, this idea demonstrates a naïve understanding of enterprise demands, for not even the best of automated systems can anticipate the special needs of each enterprise. These "black boxes" offer only a few, if any, tuning options for relevancy and do not reveal how the results were generated. In stark contrast, Autonomy's technology provides the best of both worlds, automatically retrieving the most accurate results using its conceptual understanding of content and also offering the flexibility to modify the relevancy algorithm if needed. The computational process is fully transparent to the administrator and Autonomy reveals the basis for its determinations through easily understood representations such as dominant terms and idea distances.
Both system administrators and business users are provided with a full workbench to control and tune the relevancy of search results. Some unique advantages offered by Autonomy include:
WYSIWYG (What You See Is What You Get) user interface
The weight of virtually every field (e.g. title, author) can be manipulated and many operators are available to alter relevancy
Full support for voting and document rating. End users can rate the usefulness of specific documents, and this information is subsequently used to calculate relevancy or primacy. The voting can be limited to those people whose profiles show a strong match to the subject of the document
Access to an extensive range of pertinent information, including common queries, misspellings, query types – all presented in friendly visuals
Full support for business modulated result "sponsoring" or "placement." For example, a business user can elect to promote a certain result, set of results or object (such as an advertisement) to a defined position within a result set in response to a given query or input. If a user queries for "yellow Prius," the administrator can define a rule to return the same set of results as a query for "gold Prius," with a link to the advantages of hybrid cars being the first on the returned list
Autonomy Collaborative Classifier module, which creates a workflow in which the subject matter experts and knowledge engineers, as identified by the organization, collaborate in real-time to create, modify, distribute and manage taxonomies. As the classifications are created and managed by the people who actually use them, information is organized in ways that are specific and germane to the organization
Protects user privacy by respecting entitlement rights and separating the administrator role from the "super-user", thereby ensuring the administrator will be restricted from information they are not privileged to view
In addition to providing administrators with comprehensive tools to alter the relevancy modeling, Autonomy is transparent in the methods it uses to arrive at such results. Autonomy uses the full text of the document to determine relevancy, and even with no manual configuration, administrators and users can easily understand how the results were selected. Autonomy uses many ways to justify relevancy, these include:
% relevancy: Provides percentage similarity of the document to the query
Automatic highlighting: Highlights key terms/concepts within the document
Ideas cloud: Presents list of concepts present in the results list, with variable font size and boldness to represent the number of results that include that concept
Cluster tree: Displays hierarchy of relevant entities (concepts or metadata) extracted from the results list; it displays the count of documents that contain that entity
Automatic summarization of content: Demonstrates the key concepts extracted, which may come from different parts of the document
Query journey: Delineates the logical path that IDOL took to arrive at a given set of results by revealing the key concept terms that were found, along with pertinent metadata, repositories searched and the relevancy threshold reached
Query bread-crumbing: Traces all aspects of a user’s query interaction
Autonomy enables an entire range of information processing options, both manual and automatic. The system can be configured to support as much or as little manual involvement as necessary, ensuring that Autonomy is not a "black box" where the running of the technology cannot be seen or adapted by administrators.
Summary: ...inevitably demand new compliance initiatives and increased information visibility, transparent document and records management will continue to be increasingly important to the pharmaceutical company as it continues to develop more new drugs. However, after deploying HP TRIM, the organisation’s European...
Summary: ...every year and slowing down the grant submission process, which often have short time-frames.” The solution Through interactions with ISU faculty and staff, OSPA determined that an electronic, Webbased internal proposal data form was the single most important process improvement it could implement....
Summary: ...control costs, and be compliant with applicable laws and regulations, companies must periodically purge records they are no longer required to keep. At the same time, however, deleting records entails risk. It can be challenging to determine which records—if any—may be purged, and when. KPMG, HP simplify...
Summary: ...process. However, because of Introspect’s de-duplication features, the firm produced a more consistent production set. LaCour elaborated, “In the ‘old way’ of reviewing, several reviewers could be faced with the same document and make differing determinations. Using Introspect, and our tiered...
Summary: ...in the system simultaneously, regardless of format or repository, including rich-media such as audio or video files. In order to overcome the challenge of duplicate data, it would also need to be able to identify links between related data based on more than a simple keyword match. Finally the system...
Summary: ...these requirements by simply deploying piecemeal technology. However, whereas these partial, on-thespot solutions may provide some minor benefits in the short-term, in the long run they can only serve to fragment the people and communities whose close cooperation is vital in order to achieve greater transparency....
Summary: ...search and retrieval tools across all structured and unstructured data throughout the global enterprise. Top benefits of IDOL for EnergySolutions: • Harvests data into a homogenous system without exporting documents from old systems • Processes and analyzes data from older, proprietary database systems...
Summary: ...Radioplayer - Autonomy Case Study. Based on advanced pattern matching and concept extraction, connecting to over 400 content repositories and supporting over 1,000 file formats, IDOL was the only solution able to automatically extract and form an understanding of the key concepts found in the live stream...
Summary: ...Archiving and data protection | IT case study | Bon-Ton Stores | HP. The team has configured HP Integrated Archive Platform to migrate messages older than 60 days from email servers to a central archive, index the full text of messages and attachments, and purge stored messages older than two years. Archiving...
Summary: ...on track.” Phil Smart, IT Director, Tupperware N. A. “HP staff were technically aware and capable of understanding the data volume we had, and the restrictions within Oracle that we faced. In fact, HP not only understood those problems—they helped me solve them,” says Phil Smart, Tupperware’s...
Summary: ...that they reside, so users can find what they want quickly and put it to use immediately. Using HP Autonomy’s solution, Felco users can access brand materials almost instantly, eliminating the need to wait for a disc to arrive in the mail. The HP Autonomy solution also allows Felco to publish automatically...
Summary: ...Slate tablet devices to allow users mobile access to the SMA information. This was later extended to users own iPads, and alerting to Android devices. A half-day training session was held for all users. Benefts Helping address the ‘needle in a haystack’ problem The trial was quickly judged a success,...
This is a small selection of the Autonomy case studies available, please visit our publications site at http://publications.autonomy.com/ for more information.
Summary: ...Autonomy Document Management Server Technical Brief. Relevance Ranking In evaluating all types of queries, IDOL Server employs complex algorithms based on a combination of Information Theory and Bayesian methods to automatically weight and rank the document returns by statistical relevance. Automatic...
Summary: ...and, thus, store less data on disk. StoreOnce also uses algorithms called Sparse Indexing and Contained Matching to reduce the number of times the deduplication engine has to read the data to determine if chunks match. Instead of reading an entire data chunk, these algorithms preview parts of it and compare...
Summary: ...Auminence Product Brief. MEANING BASED HEALTHCARE Auminence Based on Autonomy’s unique Bayesian inference technology, Auminence represents a breakthrough for healthcare providers seeking to transform healthcare delivery through a deeper and more meaningful understanding of their data. At its essence,...
Summary: ...processing load on the hardware on which it’s hosted. Many software solutions use an inefficient deduplication process that reads each entire data chunk on disk to determine if a new chunk is a match. This laborious process taxes the CPU and slows down hardware and other applications (Figure 2). This...
Summary: ...connects transparently with MOSS to provide information governance, pan-enterprise search and analytics for SharePoint. At the heart of Autonomy’s solution lies the Intelligent Data Operating Layer (IDOL). Using complex pattern-matching algorithms and probabilistic modeling, IDOL forms a conceptual...
Summary: ...extracted by IDOL server. The underlying algorithm is the same regardless of the language used. While Autonomy's technology is language independent, it can be beneficial to use language dependent features in order to optimize IDOL server’s ability to match concepts irrespective of their appearance in...
Summary: ...objectives. In order to determine the primary entities within a story, entities can also be compared with other concepts covered within the document and a relevance score calculated. A higher relevance score would mean that the entity is more central to the piece of information. Product Brief Key Benefits...
Summary: ...global enterprise. IDOL’s language-independent technology develops a statistical understanding of the patterns of any language using sophisticated probabilistic modeling and patternmatching techniques. It currently supports over 100 languages and provides cross-lingual search. Autonomy also provides...
Summary: ...DTCC, the industry’s preferred matching and confirmation utility, ensures seamless, automated processing of commodity derivatives documentation. Autonomy’s Intelligent Data Operating Layer (IDOL) is a sophisticated mathematical and statistical approach to information that alters the way computers...
Summary: ...commodity derivatives documentation. Autonomy’s Intelligent Data Operating Layer (IDOL) is a sophisticated mathematical and statistical approach to information that alters the way computers relate to electronic information, creating a new paradigm of information technology and solutions. Transparent...
Summary: ...Real-Time Predictive Query (RPQ) RPQ anticipates what the user would want to know and dynamically generates and recommends the results list as the user is entering the query. These results dynamically change as the user enters more text into the query. RPQ combines popular query terms with knowledge about...
Summary: ...HP Autonomy Consolidated Archive and StoreAll (brochure). Using probabilistic modeling and advanced pattern matching to form a conceptual and contextual understanding of ESI, the solution delivers advanced functions such as automatic classification, end-to-end e-Discovery, and advanced capabilities that...
This is a small selection of the Autonomy Product Briefs available, please visit our publications site at http://publications.autonomy.com/ for more information.
Summary: ...even people’s interests. Bayesian Inference Thomas Bayes was an 18th century English cleric whose work has become a central tenet of modern statistical probability modeling. Bayes' efforts centered on calculating the probabilistic relationships between multiple variables and determining the extent to...
Summary: ...to Bayesian Inference and Claude Shannon¹s Principle of Information. Bayes Theorem has become a central tenet of modern statistical probability modeling. Autonomy’s advanced pattern-matching technology exploits high-performance probabilistic modeling techniques and extract a document's digital essence...
Summary: ...to be encountered. Bayesian Inference The theoretical underpinnings for HP Autonomy’s technology approach can be traced back to Thomas Bayes, an 18th century English cleric whose work has become a central tenet of modern statistical probability modeling. Bayes’ efforts centered on calculating probabilistic...
Summary: ...of information are automatically linked via dynamic probabilities. The second reason is that the documentmatching algorithm itself within IDOL uses widespread “short-circuiting” and iterative calculation to ensure that it only performs exactly as much calculation as is required. In essence, the key...
Summary: ...summarization, taxonomy generation, clustering, eduction, profiling, alerting and retrieval. Autonomy’s IDOL server is built upon the seminal mathematical works of Thomas Bayes and Claude Shannon, as well as on a range of innovations that are covered by 170 patents. Autonomy technology identifies the...
Summary: ...as quantum clustering, to form them. It understands that there is an X probability that the content in question deals with a specific subject by studying the preponderance of one pattern over another. IDOL is built on algorithms rooted in Bayesian Inference and Shannon’s Information Theory. With Bayesian...
Summary: ...agnosticism at a fundamental level. The Intelligent Data Operating Layer that lies at Autonomy's heart uses a unique combination of Bayesian Inference and Shannon's Information Theory to apply a technique known as Adaptive Probabilistic Concept Modelling* to the problem of understanding unstructured information....
Summary: ...security to Fortune 500 companies. The Foundations of IDOL Autonomy's strength lies in a unique combination of technologies that employs advanced pattern matching techniques, using Bayesian Inference and Claude Shannon's principles of information theory.
...
Summary: ...across the enterprise, even if it is not in the same language as their query. How do I search rich media? Current trends anticipate the rapid proliferation of rich media, widespread adoption of VOIP, growing use of IPTV and increased scrutiny of white collar crimes. This widespread adoption of rich media...
Summary: ...sequence A given W – the Acoustic Model By combining these models and using Bayes’ principles, Autonomy identifies the word string W as having the highest probability given the acoustic sequence: This model provides the probability of a word sequence occurring and the statistical relationships between...
Summary: ...over the access control rights, which remain a black box to the server itself. Company intranet and intelligence environments are characterized by mandatory access control where access to specific documents is determined by a user’s clearance level and status of the document. Desktop Search In theory,...
Summary: ...engagement and driving revenue. Successful organizations will adopt an approach that leverages proven mathematical techniques and algorithms of inference and statistical analysis to extract meaning from both structured and unstructured information and make it available throughout the customer lifecycle,...
This is a small selection of the Autonomy White Papers available, please visit our publications site at http://publications.autonomy.com/ for more information.
Related Case Studies
top of page
RSS Feeds
Find us on Facebook
Follow us on Twitter
See us on YouTube
Find us on LinkedIn
