Artezio Big Data Technology Profile

Transcription

1 Artezio Big Data Technology Profile Artezio is an ISO 9001:2008 certified software development company. Since 2000, Artezio has completed over 1000 projects for its international corporate customer base, thus the Company has become one of the key players on the European/US software development market. Artezio Inc. www. Skype: artezio_software

2 Company Overview Established in 2000 Development centers in Belarus (Minsk, Vitebsk, Mogilev), Russia (Moscow, Nizhniy Novgorod, Saratov), Ukraine (Kharkiv) and Latvia (Riga) Sales and Operations Offices in the US: Boston (HQ), New York, Florida, California ISO 9001:2008 certified Operations in Europe, the US, Middle East and JAPAC Over 1000 successfully accomplished projects worldwide Delivery excellence confirmed by trusted international ratings: Global Outsourcing 100, Global Services 100, the Black Book of Outsourcing, etc. Artezio Inc. www. Skype: artezio_software

3 Expertise Database analysis and BI - Pentaho - Oracle BI NoSQL databases - Cassandra - MongoDB Intellectual Data Processing - Apache Solr - Apache Mahout Distributed Compilation - Apache Hadoop - Apache ZooKeeper Artezio Inc. www. Skype: artezio_software info@

4 Products Artezio Inc. www. Skype: artezio_software

5 Artezio has developed a system called Sea-ART to solve the problem of advanced information search in the global network. It helps to handle the following tasks: 1. It enables a user to form flexible queries, wherein the following can be used as input information: a text row a formatted text a file a link to a resource with the contents similar to the required 2. It allows carrying out trainings of search campaigns: a user sorting out intermediary search results in a manual mode can specify subjective relevance of each found result to improve its quality at each subsequent search iteration. 3. It makes it possible to operate search campaigns including: view of history search results (cached copy of the result) view of direct links to the found sources simultaneous running of several campaigns classification of search results including elimination of undesirable categories 4. It contains a notification on location of the required information. The core of the system is a search platform Apache Solr. In particular, the modified under the project needs Bayesian algorithm of text classification, which proved to be one of the best probabilistic methods of comparison of any data files including texts. A search mechanism gets a draft information flow on the first step: Google search results Corporate communications Social network news feed Other issues Artezio Inc. www. Skype: artezio_software info@

6 Case Studies Artezio Inc. www. Skype: artezio_software

7 Case Study: Data Repository of the Network Traffic for its Further Processing and Analysis Project Goal To develop a system that allows users to: quickly download a large amount of data of various structures into a database aggregate data for its display in BI systems provide quick data access Challenge Data on the network traffic is stored in files generated by the system. Detail levels are set by a user, therefore their structure can be different. Data can be of a large size and generated very fast. Technologies NoSQL MongoDB This DBMS allows downloading quickly a large amount of data with an uncertain structure as well as processing it and providing quick data access. Data loading into the database was implemented by an external program. Results The system that met the set requirements was delivered to the Customer. Artezio Inc. www. Skype: artezio_software info@

8 Case Study: Flexible CRM Project Goal The main aim of this project is to develop a CRM system; the users-administrators of the system can create and customize the app interface and business processes for their own business tasks, without addressing IT companies. This system will allow clients to quickly receive the necessary software to provide workplace for the company employee as well as improve business performance and quality of the services provided. Challenge A large number of users of the flexible system generate a big amount of data with an uncertain structure. The data can be of various types, and it is necessary to save it and grant quick access. Technologies It would be more convenient to use NoSQL databases to store business data with an uncertain structure, but it doesn t not guarantee maximum security of the business critical data. For this reason, traditional relational databases (Oracle, MS SQL Server, PostgreSQL, MySQL) were used with a special table structure. NoSQL MongoDB was used to store larger amounts of less critical business data. For example, application logs, chat history, history of data editing, etc. The app itself was developed using Java. Results A CRM prototype was implemented that allowed users to configure the software for employee s work by their own, without engaging any IT engineers. Artezio Inc. www. Skype: artezio_software info@

9 Case Study: BigData Management Platform as Cloud Service Project Goal To build scalable and reliable Big Data management solution from scratch. Description The Data Management Platform was decomposed into several key components: Metadata Storage, RAW data Depositary, Text Classifier module, Web Crawling module. Each of these components provided its own RESTful API, implemented as Jersey-based web application, running on Tomcat server. The Metadata Storage provides API to manage relatively small amounts of rarely changeable high-level client data such as taxonomies, data endpoint descriptions and async job statuses. The client data was persisted in PostgreSQL database that simplifies supporting of sensitive data consistency based on constraints and transactions. The RAW data Depositary provides API to transfer of large amounts of data between various sources and destinations, including WebDAV, HDFS, FTP and SSH locations, via Akkabased async jobs. It also collects data transfer statistics and exposes corresponding REST API. The Web Crawling module encapsulates functionality to retrieve and pre-process content of web pages to create learning datasets for ML algorithms. The Text Classifier module exposes API to define Learning data sets, managing async ML jobs and performing arbitrary texts classification. Definitions of Learning data persisted in PostgreSQL to ensure their consistency. Async ML jobs are implemented as map-reduce wrappers around Weka text classification algorithms, and executed in Hadoop over HDFS storage, when all the intermediate and final results are stored in Aerospike NoSQL database. A SoapUI-based integration tests suite were introduced to ensure stability of all of the exposed REST APIs. Technologies Linux Java Spring Jersey Akka Hadoop/HDFS Results All the key components of the Data Management Platform were deployed on production environment on time and are capable to handle all the declared usage scenarios. Artezio Inc. www. Skype: artezio_software info@

10 Case Study: Mining for data in a large database Project Goal To develop a system for identification of multiple accounts belonging to one subscriber. Description For developing the database application Artezio used C++ programming, the CORBA technology (the omniorb CORBA broker), Oracle stored procedures and Oracle OCI. Artezio implemented the concept of rule-based search where fields do not have to be absolutely identical to match, but more or less resemble the sought combination of the search request parameters. Original documentation with detailed technical requirements was provided by the customer. It was refined and updated during the analysis and design phase by Artezio system architects. Artezio wrote original drafts of user documentation including an installation guide and a user guide. All coding and testing was done at the Artezio SDC in Moscow. The entire project was managed by an Artezio project manager who reported to the customer s on-site coordinator. Technologies C++ CORBA (omniorb broker) Oracle 9i Oracle Call Interface Apache HTTP Server Python 2.2 Microsoft Windows 2000/XP Linux Red Hat 7.2 SUSE Linux 9.0 Professional HP-UX 11i Plug-ins Scope 40 man-months Results The prototype system version was delivered on schedule. On the prototype stage Artezio demonstrated that the system was capable of providing tangible query results in a reasonable time. Then the project entered another round of planning, design, and development. Within five months a full-featured version of the system was released. Artezio Inc. www. Skype: artezio_software info@

11 Case Study: Artezio helps to collect and analyze visitor statistics for a health care portal Project Goal The challenge was to create a Web-based interface which would present portal activity statistics and improve user experience. Description Artezio engineers designed layouts of Web pages one page for each report type. After a few interactive conferences with the client, the final page design was approved. Artezio team created HTML templates using Java Servlet/JSP technology,which allowed a dynamic HTML pages generation. Visitors activity statistics and reports were imported into an Oracle database. The imported data was transformed into a structured format for further analysis and report generation: both data transformation and reporting were implemented based on Oracle stored procedures. The customer emphasized the need to generate custom reports based on users preferences. To accomplish this, Artezio engineers developed customizable, hierarchical, tree-view structures on the front-end for easy navigation and statistics review. In addition to the Web application, Artezio delivered an installation utility which made it easier to deploy and customize the application. Technologies Java 2 Enterprise Edition v.1.3 JSP, Java servlets JetBrains IntelliJ IDEA 3.0 BEA WebLogic Server 7.0 Oracle 8i Enterprise Edition Microsoft Windows 2000 Scope 1,5 man-months Results The product developed by Artezio has made it simpler to track occasional browsing problems, helped to proactively modernize the site and optimize it depending on combinations of browsers/ operating systems used by subscribers, which is important these days with the arrival of a great number of technology platforms on the market. Artezio Inc. www. Skype: artezio_software info@

12 Case Study: Data matching module for a credit card issuer Project Goal To develop and implement a module that will help reduce service costs and improve the quality of the operational data. Description Artezio staff developed the technical specification, technical design, and test plan documents for the project. Also, Artezio created a number of use cases to describe the behavior of the matching process as well as test cases to describe testing procedures in detail. To meet the Customer s needs Artezio developed the Matcher module both as a standalone JAVA application and as a Windows service. This module is easily pluggable into a data processing system via JMS. The application was based on a multi-layered component architecture. The main part of it rule engine was implemented to perform data matching and data verification. The application was designed to use multiple scenarios, filters, data pre- and post-processors, comparison algorithms, and it could be easily reconfigured at runtime by a non-technical employee. In addition, it was enhanced with recovering and logging mechanisms. Technologies JMS Spring framework Jencks JCA container ActiveMQ TestNG Log4j Scope 1.5 man-months Results About 84% of the incoming data was successfully processed by the automated Matcher, which is 4 times greater than the Customer expected. Artezio Inc. www. Skype: artezio_software info@

13 DBA and BI Team 5 certified professionals with 15+ years of experience in IT business (Database, Warehouse and Data Mart areas, as well as BI systems) The team participated in various Big Data related projects (RDBMS, BI, ETL, NoSQL database) Java Team More that 150 OSJP certified professionals Developed more than 50 high-loaded Big Data related systems Complete JEE stack Analysis Team More than 15 system analysis professionals with proven track record in requirements engineering, business rules and enterprise data analysis, including experience working with high loaded systems. Latest experience includes analysis provided for government procurement Web-portal and worldwide education network. Artezio Inc. www. Skype: artezio_software info@

14 Artezio thanks for your attention Contacts Web-site: www. Skype: artezio_software Artezio Inc. www. Skype: artezio_software