The world's largest and most secure intelligence organizations have deployed Autonomy's Intellectual Asset Protection System (IAS) to safeguard their most sensitive information assets. Autonomy provides all aspects of security management, including front-end user authentication, back-end entitlement checking and secure encrypted communication between the IDOL Server and its client applications with 128-bit Block Tiny Encryption Algorithm (BTEA). IDOL's mapped security model is the only empirically proven index security model that scales in the enterprise.
There are three general security models currently available:
1. Unmapped Security
Unmapped security is the traditional method used by source repositories and search engines. For every potential match to a given query, a call is made via the native repository's API (e.g. Documentum) to ascertain the access privileges for that particular document. A single query consequently bombards the native repository with document privilege requests as the retrieval system attempts to assemble a relevant results list from thousands of candidate hits. This method presents significant performance and scalability problems.
Unmapped Security
Mapped Security
"Security is a key differentiator for IDOL. IDOL offers "mapped security" and near real-time synchronization of security entitlements with source content repositories - making it a great fit for highly secure search scenarios"
The Forrester WaveTM: Enterprise Search Platforms, Matthew Brown
Autonomy recommends mapped security but also offers the choice between mapped, unmapped and a hybrid of both. Autonomy also supplies plug-in sample code, so that customers, OEMs and partners are able to develop and implement their own form of security plug-in.
2. Cached Security
Cached security is the method of choice for legacy systems. Cached security marginally relieves the scalability problem of unmapped security by storing results for queries it has already seen. Consequently, when a user repeats a query, the result set can be retrieved from the cache rather than triggering a network-mediated request. However, this approach still relies on calling out across the network directly to the repository for each new query. In addition, it also misses potential results, as the result sets stored within its memory do not dynamically update new information.
3. Autonomy's Unique IAS Mapped Security
Only Autonomy offers mapped security - a highly configurable, secure, accurate, and fast method for respecting third party security entitlements. IDOL maps the underlying security model in the form of ACL, group, role, protective markings, etc. from all of the underlying repositories directly into the kernel of the IDOL engine itself, and stores the information in an encrypted field. As a result, IDOL does not need to send any requests across the network to the data stores when building up a results list. What the user is allowed to see is assessed "inline" within the IDOL kernel at speeds that exceed the response times of the native repository. Unlike other techniques, the security model is never out of date as the transitional signaling mechanism within the connector layer informs IDOL in real-time of any updates or changes to permissions within the underlying content.
Since IDOL's architecture is inherently modular by design, it requires multiple subsystems to communicate with each other, often across insecure networks. All communication between these processes may be encrypted (Secure Sockets Layer), so that packet sniffers who are able to break past a firewall are unable to read the content of traffic between IDOL modules. All of the system's modules are capable of operating in a secure communications mode providing, at minimal processing overhead, the protection of 128-bit encryption. Additionally, IDOL can leverage SSL for both aggregation and querying of content, including access to SSL encrypted sites.
"Autonomy fully meets the stringent requirements of managing records in a classified environment where security and access are critical."
IT Business Net, 2011
Scalability and Performance
The management of structured and unstructured content requires a platform that can meet the most rigorous performance requirements and be easily resized commensurate to business needs. IDOL scales to support the largest enterprise-wide and portal deployments in the world, with presence in virtually every vertical market. Since IDOL's scalability is based on its modular, distributed architecture, it can handle massive amounts of data on commodity dual-CPU servers. For instance, only a few hundred entry-level enterprise machines are required to support ChoicePoint's 10 billion record footprint. By comparison, a competitor uses 150,000 machines to handle the same amount of data.
A single IDOL engine can:
Support an estimated 30 million documents on 32-bit architectures and over 250 million on 64-bit platforms
Accurately index in excess of 60 GB/hour with guaranteed index commit times (i.e. how fast an asset can be queried after it is indexed) of sub 5ms
Execute over 2,000 queries per second, while querying the entire index for relevant information, with subsecond response times on a single machine with two CPUs when used against 30 million pieces of content
Support hundreds of thousands of enterprise users, or millions of web users, accessing hundreds of terabytes of data
Save storage space with an overall footprint of less than 30% of the original file size
This enhanced scalability results in hardware cost-savings as well as the ability to address larger volumes of content. Though IDOL scales extremely well on commodity servers, its flexible architecture can take full advantage of massive parallelism, SMP processing capabilities, 64-bit environments (such as Intel Itanium 64-bit architecture), software platforms (such as Solaris 10, Linux 64, Win64, etc), distributed server farms, and all common forms of external disk arrays (i.e. NAS, SAN etc) to further improve performance. This flexibility extends to being able to leverage one or a combination of these different environments.
How It Works
Content from various repositories is aggregated by connectors and then indexed into the IDOL Server or for dissemination across multiple IDOL Servers, through the Distributed Index Handler (DIH). The DIH can efficiently split and index copious quantities of data into multiple IDOL Server instances, optimizing performance by batching data, replicating all index commands and invoking dynamic load distribution. The DIH can perform data-dependent operations, such as distributing the content by date, which allows for more efficient querying. Performance is augmented by the Distributed Action Handler (DAH), a distribution server that allows the user to distribute action commands, such as querying, to IDOL Servers. Multiple copies of IDOL Servers, to which the DAH propagates actions, further ensure uninterrupted service in the event of server failure. For flexibility, both the DAH and the DIH can be configured to run in mirroring mode (IDOL Servers are exact copies of each other) and non-mirroring mode (each IDOL Server is configured differently and contains different data). In addition, the Distributed Service Handler (DiSH) component allows effective auditing, monitoring and alerting of all other Autonomy components.
Linear Scalability
Performance and capacity can be doubled by simply replicating the existing machine. This allows scaling predictions to be made without worry about bottlenecks.
Load Balancing
Data is automatically replicated across multiple servers and user requests are load-balanced across these replicas, guaranteeing performance, reducing latency and improving user-experience.
Mirroring / Failover
Automatically generated replicas are used to provide a pool of servers, the primary resource is automatically selected and the system switches to secondary systems if it fails so that service continues uninterrupted.
Distribution
For organizations that are geographically distributed, local replicas are automatically created and utilized where possible. Remote copies are only used when a local system fails, thereby building fault tolerance whilst maintaining the benefits of local performance and a reduction of resource overhead into a single, seamless service.
Adaptive Probabilistic Concept Caching
Frequently used concepts are maintained in memory and query results are returned as quickly and efficiently as possible.
Multi-dimensional Index & Query Throttling
By using a multi-dimensional index to provide valuable information to the distribution components, IDOL precludes bottlenecks and unbalanced peak loads during the indexing and query process.
Autonomy provides prioritized throttling based on:
Time: maximize index/query performance based on the time of day (i.e. work hours)
Location: prioritize activity based on the server landscape
Status: arbitrarily assign prioritized status for processing
"We have worked with Autonomy for a number of years due to their ability to offer a next-generation enterprise search platform that doesn't necessitate a trade-off between performance, security and scalability."
Mr. K. Sriram, Senior Vice President, Satyam Consulting and Enterprise Solutions Practice
Instruction-Level Parallelism
IDOL programmatically expresses itself as an expanding collection of operations. These operations can and are executed in serial pipeline form yet the inherent logic of simultaneously processing disparate forms of unstructured, semi-structured and structured data requires a high degree of parallelism. Not only does IDOL need to ingest multiple streams and types of data, it must also provide a real-time answer or decision against that data as it is indexed rather than force the user to wait an arbitrary period until serially accessed resources becomes available.
As a consequence IDOL has been designed with instruction-level parallelism (ILP) as the core of its process and operation model. ILP by definition is limited by the serial instruction model of scalar processors and thus Autonomy has been an extremely conscious early adopter of all forms of parallel architecture from multi-CPU, hyper-threading and now single die multi-core processing.
The engine's default process model is multi-threaded (using a configurable number of threads). IDOL operations can either be grouped by class, with indexing and querying performed by separate threads or for n-core models a single operation can be "atomized" into multiple threads. Concurrent querying and indexing is the default with no requirement whatsoever for "locking" any part of the indexes while querying takes place. All major multi-core manufacturers are supported, including Intel, AMD and the latest Niagara offerings from Sun Microsystems.
Classic scalar models that rely on Moore's predicted doubling of transistor density over 18 month intervals have already demonstrated wire and memory access latencies in addition to heat sealings. As a result, hardware manufacturers such as Intel have declared multi-core strategies as key to crossing the consumer "teraflop" threshold and aim to produce n-core 32 billion transistor die within the next 10 years. Autonomy is actively pursuing a Tera computing R&D simulation program in anticipation of increasing transistor and core density and the declared aim of such manufacturers. Autonomy is currently performing "coalition" simulations of split thread IDOL operations against n-core "battalion" processor units that blend general-purpose cores with more specialist cores such as those dedicated to signal processing. These blended core units are predicted to be the first consumer teraflop chips. Autonomy is developing process thread models that dynamically co-opt different core types to act in "coalition" to perform the simultaneous deconstruction and analysis of unstructured sources such as video that combine visual and auditory attributes.
Forthcoming Events for Security, Scalability and Performance
Archived Events for Security, Scalability and Performance
Summary: ...capacity − All instances support archiving from multiple database technologies • Web console − Linux operating system running on a virtual server − Current installed capacity: 20 CPU, 40 GB RAM • Repository database − Oracle 11gR2 running on Linux operating system • Target (history) databases...
Summary: ...across the globally distributed organization ■ Inconsistent, ad hoc profi ling made content difficult to find ■ Large volumes of unmanaged email content caused email overload, undermined centralized management, and increased risk Benefits of Using Interwoven ■ Remote access and caching capabilities...
Summary: ...assets across the globally distributed organization • Inconsistent, ad hoc profiling made content difficult to find •Large volumes of unmanaged email content caused email overload, undermined centralized management, and increased risky Benefits of using Autonomy WorkSite •Remote access and caching...
Summary: ...and unstructured data located in systems from the acquired financial institution Benefits • Located and collected data from over 70,000 systems distributed geographically • Pre-Culled and deduplicated the over 500 TB of data • Provided a secure long-term archive environment that supports legal hold...
Summary: ...where a message might reside. Searches done in hours instead of days For the IT team, it’s much easier to produce messages on demand. “An area where HP Integrated Archive Platform has gone beyond expectations is the ease of use in doing searches and building queries,” Taylor says. “We have two...
Summary: ...of data. It has now been able to reduce its storage requirements by more than 80% after implementing Connected Backup. “The solution support provided by HP Autonomy has been extremely effective, with the company adhering to agreed SLAs and responding to any queries very promptly.” —IT Manager, Glassolutions...
Summary: ...the integrity of Dongkuk’s business and transaction data is protected, and should there be a service interruption, the data can be rapidly restored to ensure business continuity. In addition, because the software is designed to minimize impact on server CPU and I/O, it reduces the risk that back-up...
Summary: ...huge part of the way brown Rudnick works, and will make the new solution a powerful addition to our culture.” In addition, WorkSite’s collaborative features will help staff manage critical dates and resolve issues centrally, enabling distributed teams to securely share client case files, memoranda,...
Summary: ...center and the second AiO system is based in the backup data center. The systems are equipped with HP Storage Mirroring Software, which provides asynchronous data replication between the sites. “We spend about 30 percent less time managing storage then we did before,” Altynikov notes.
...
Summary: ...distributed authoring. “In a large organization you have to accept that you can’t have everything coming through one department because it causes a bottleneck. Yet, if you allow distributed authorship you must have workflow that controls approvals. TeamSite is quite effective at setting this up. It...
Summary: ...world’s largest organizations Industry Consulting Objective Tape-based backups were plaguing Roundarch with lengthy backup processes, and with data doubling year over year backup windows could no longer be efectively met. The company needed an automated, cloud-based backup solution that could manage...
Summary: ...of a file system from 78 hours down to 10 ho MIHS is able to use HP Data Protector and the HP StoreOnce B6200 to backup data across its physical and VMware environments. This includes both Microsoft Exchange and SQL servers, as well as its Epic medical records system, which is based on a Cache database....
This is a small selection of the Autonomy case studies available, please visit our publications site at http://publications.autonomy.com/ for more information.
Summary: ...well on commodity servers, IDOL’s flexible architecture can further improve performance by taking full advantage of massive parallelism, SMP processing capabilities, 64-bit environments, software platforms (such as Solaris 10, Linux 64 and Win64), distributed server farms and all common forms of external...
Summary: ...modular architecture, it can handle massive amounts of data on commodity dual-CPU servers. IDOL delivers linear scalability through a multi-threaded, multi-instance approach with load-balancing to intelligently distribute the indexing and query workload. Mapped Security The biggest single constraint on...
Summary: ...user authentication, back-end entitlement checking and secure encrypted communication between the IDOL Server and its client applications with 128-bit Block Tiny Encryption Algorithm (BTEA). There are three security models offered within the market today – “Unmapped,” “Cached” and “Mapped”...
Summary: ...Stack App App Xen Hypercall API Xen Hypervisor Hardware Guest OS Guest OS Application Application Operating System Virtualization Layer Host Operating System Intel®Architecture CPU Memory NIC Disk MS Virtual Server app 1 app 2 HP‑UX 11i v2 Linux RH4 app 1 Windows app 1 app 1 Linux RH4 Windows app 1...
Summary: ...machines over the Internet or private network connection. Organizations can manage global backups from a single console with features such as grouping of servers for distributed administration, role-based administrations, and enterprise reporting. A simple agent can be installed on one or many machines...
Summary: ...are using virtualization to consolidate servers and reduce infrastructure costs. As the number of virtual machines grows, it becomes increasingly difficult to manage the backup process for virtual machines alongside the existing physical machines. A complicated backup management solution only increases...
Summary: ...minimizes the load on CPU processing and maximizes application availability. It uses as little as a tenth of the memory of competing solutions, which means it can be deployed on application or backup servers, and even virtual machines, without crippling performance.4 Data Protector software deduplication...
Summary: ...and recovery performance. HP StoreOnce’s thin, efficient footprint minimizes the load on CPU processing and maximizes application availability. It uses as little as a tenth of the memory of other available solutions, which means it can be deployed on application or backup servers, and even virtual machines,...
Summary: ...connection for uninterrupted remote data protection. Data is moved offsite to secure, mirrored data centers and is completely secure and protected at every step of the way using stringent procedures, protocols, and standards. Autonomy LiveVault encrypts all data at the source using 256-bit AES encryption...
Summary: ...Requirements Platforms Supported: • Microsoft Windows NT, Windows 2000 & XP • Linux • Sun Solaris • Any other POSIX compliant of UNIX on request Minimum Recommended Server Specifications: • Pentium CPU (Intel or Intel Compatible Processor) • 128 MB RAM J2EE-compliant application server support:...
Summary: ...LiquidBPM Engine is the core component of LiquidBPM. Its numerous features include caching, thread-pooling, persistence, clustering, load balancing, notification and scalability. The LiquidBPM Engine’s ability to automate processes improves efficiency and increases throughput by enforcing predictable...
Summary: ...location, as soon as it becomes available. Active folders can be trained by example, such as by dragging and dropping a document into the folder or by natural language query. Multiple data formats are available in a single folder, and results are downloadable for offline viewing. Enterprise Class Security...
This is a small selection of the Autonomy Product Briefs available, please visit our publications site at http://publications.autonomy.com/ for more information.
Summary: ...intervals have already demonstrated wire and memory access latencies in addition to heat sealing’s. As a result, hardware manufactures such as Intel have declared muti-core strategies as key to crossing the consumer “teraflop” threshold and aim to produce n-core 32 billion transistor die within...
Summary: ...and a reduction of resource overhead into a single, seamless service. • Load Balancing: Data should automatically be replicated across multiple servers and user requests should be load balanced across these replicas, guaranteeing performance, reducing latency and improving user-experience. • Mirroring/...
Summary: ...is that mapped security mitigates this security challenge by respecting third-party security entitlements without either caching or using lastminute query time checks against the native repository. 6 Enterprise Search: Addressing Security and Entitlement Issues Ability to Choose Autonomy recommends mapped...
Summary: ...versus manual partitioning, the bottom line seems to be that even on the new 64 bit architecture with 4 screaming Intel processors, and SQL 5 - the upper limit of MOSS’ content repository is 500GB. Assuming allocation for index data and other overhead this really allows less then 20 million objects...
Summary: ...Distributed Environments EAS has a unique Parent/Child architecture that allows the archiving process and the archive service to be established in the optimum locations after such factors as network bandwidth, CPU consumption, anticipated demand, and archive policy have been accounted for.
...
Summary: ...to configure the cache, and control aspects of its behavior. • Thread pooling. Threads are used by the Cardiff LiquidBPM Engine to analyze workfows. They are also used to execute tasks in a workfow that can be done concurrently. Threads tend to be an expensive computational resource. The engine utilizes...
Summary: ...CPU hardware—same as above (1GB of RAM dedicated to each WebLogic server) Database server hardware and software configuration n Database: Oracle 10g, deployed on a separate DB server machine n Database server OS:Windows 2003 Server n CPU: Intel Xeon 3GHz, 4 processors, 2 GB RAM III. Benchmark Results...
Summary: ...searched as easily as text Understands words in context for unrivalled accuracy Patented search technology drastically reduces CPU and memory requirements Automatic customization via additional text material Fixed-latency real-time mode for media monitoring operations Speech-To-Text Autonomy’s speech-to-text...
Summary: ...Autonomy Whitepaper - The Power of Autonomy IDOL. Access rights are assessed ‘in line’ within the IDOL kernel at speeds that exceed the response time of the native repository. IDOL uses 128-bit encryption and can leverage SSL for aggregation and querying of content. Autonomy’s proven technology...
Summary: ...covering all of the repositories IDOL supports. Autonomy provides all aspects of security management including document and intradocument access control against user, group and role level entitlement. Encrypted inter-machine and intra-process communication protocols are woven into the fabric of Autonomy's...
Summary: ...and can also set the encryption key. Qfiniti uses AES 256-bit encryption to encrypt the recordings and RSA-1024 encryption to encrypt the user-definable encryption key. A February 17, 2010 update to the PCI-DSS standard will require organizations to completely block (or mute) customer sensitive data from...
Summary: ...is designed to get more utilization from physical servers, CPUs, memory, and networks are consumed meeting other compute loads, leaving fewer resources available for the backup process. When choices are limited, what do you do? Do you avoid backing up some virtual machines (VMs)? Do you limit the number...
This is a small selection of the Autonomy White Papers available, please visit our publications site at http://publications.autonomy.com/ for more information.
There do not seem to be any press releases related to this page in 2011 at the moment, please visit the news section on www.autonomy.com for the latest news.
Related Case Studies
RSS Feeds
Find us on Facebook
Follow us on Twitter
See us on YouTube
Find us on LinkedIn
