MongoDB and AWS: Integrations

MongoDB and AWS Integrating with AWS Services Partner Technical Solutions, MongoDB Inc. Sandeep Parikh #mongodb

Recap: Deployment and Availability • MongoDB basics • Deployment configurations • Instance types • Best practices • Slides and recording: – http://www.mongodb.com/presentations/mongodb-and- amazon-web-services-deploying-high-availability

Recap: Storage Configurations • Storage options • Simple recommendations • Backup and restore • Advanced configurations • Slides and recording: – http://www.mongodb.com/presentations/mongodb-and- amazon-web-services-storage-options-mongodb- deployments

Agenda • Available Services • Integrations • Infrastucture • Future Directions • Questions

Available Services

AWS Services Compute Storage Persistent IPs DNS Hadoop Data Warehouse Stream processing App deployment Orchestration Provisioning App services Caching

AWS Services Compute Storage Persistent IPs DNS Hadoop Data Warehouse Stream processing App deployment Orchestration Provisioning Security Caching

Integrations

CloudFormation • Simplify provisioning and deployment • JSON-based templates • Manage like source code • Specify all manner ofAWS components • Boostrap for other tools like Chef or Puppet

"Parameters" : { "KeyPairName" : { "Description" : "EC2 KeyPair to enable SSH access", "Type" : "String" }, "SecurityGroupName" : { "Description" : "EC2 Security Group", "Type" : "String” }, "InstanceType" : { "Type" : "String", "Default" : ”m3.large", "AllowedValues" : [”m3.large”,”m3.xlarge”,”m3.2xlarge”], "Description" : "EC2 instance type" } }, CloudFormation Sample

"Properties" : { "InstanceType" : { "Ref" : "InstanceType" }, "ImageId" : { … }, "SecurityGroups" : [{ "Ref”: “SecurityGroupName" }], "KeyName" : { "Ref" : "KeyPairName" }, "EbsOptimized" : "true", "BlockDeviceMappings" : [{ "DeviceName" : "/dev/xvdf", "Ebs" : { "VolumeSize" : "200”, "Iops" : "1000", "VolumeType" : "io1”, "DeleteOnTermination" : “false” }}] CloudFormation Sample

CloudFormation Templates • https://github.com/crcsmnky/aws-cfn-mongodb • Templates to launch single-node MongoDB deployment • Each one implements our best practices – EBS-optimized, PIOPS, ulimit, readahead • Used to generateAWS Marketplace instances

CloudFormation Templates Clone the repo Upload the CF template Instance provisioning starts Instance clones repo Instance runs setup script Instance provisioned and deployed

CloudFormation Tools • https://github.com/cloudtools/troposphere • Python package to generate CF templates • Next versions of our templates will leverage this • Coming soon: Replica Sets • Coming later: Sharded Cluster

Elastic Map Reduce • Quickly deploy and run Hadoop in AWS • Tuned distributions to run on top of EC2 • Provision deployments with any number of nodes • Supports Spot and Reserved pricing for savings

EMR and MongoDB • https://github.com/mongodb/mongo-hadoop • MongoDB-Hadoop connector – Bi-directional access to/from MongoDB • Supports MapReduce, Hive, Pig, Streaming • Read/write from – MongoDB deployments or – BSON backup files

EMR with MongoDB MongoDB BSON S3 EMR EMR EMR EMR EMR EMR EMR EMR EMR EMR

EMR Workflow Bootstrap script • MongoDB-Hadoop • MongoDB Java driver Copy resources • Bootstrap script • MapReduce job Launch EMR • Instance type • Instance count • Arguments MapReduce Output • MongoDB • BSON in S3 EMR Logs • Written to S3

$ elastic-mapreduce --create --jobflow ENRON000 --instance-type m1.xlarge --num-instances 5 --bootstrap-action s3://$S3_BUCKET/bootstrap.sh --log-uri s3://$S3_BUCKET/enron_logs --jar s3://$S3_BUCKET/enron-example.jar --arg -D --arg mongo.job.input.format = com.mongodb.hadoop.BSONFileInputFormat --arg -D --arg mapred.input.dir = s3n://mongo-test-data/messages.bson --arg -D --arg mapred.output.dir = s3n://$S3_BUCKET/BSON_OUT --arg -D --arg mongo.job.output.format = com.mongodb.hadoop.BSONFileOutputFormat EMR Launch

Elastic Beanstalk • Deploy and manage applications • Handles provisioning, scaling, load balancing • Built on EC2, S3, SNS,Auto Scaling

Elastic Beanstalk Architecture App Serve r App Serve r App Serve r Security Group Elastic Load Balancer Auto Scaling Group

Elastic Beanstalk with MongoDB App Serve r App Serve r App Serve r Security Group Elastic Load Balancer Auto Scaling Group mongo s mongo s mongo s MongoDB

Elastic Beanstalk with MongoDB • Customize and configure software that your app needs (e.g. mongos) • Install packages • Create files • Execute commands (before or after app is setup) • Control system services • http://docs.aws.amazon.com/elasticbeanstalk/latest/ dg/customize-containers-ec2.html

Infrastructure

Elastic IPs • EC2 instances use dynamic IP addresses • EIPs are static addresses that can be assigned to individual EC2 instances • Unfortunately you have a limited number

Route53 • Highly available and scalable DNS service in AWS • Hostnames can be assigned to EC2 instances, ELB instances, or S3 buckets • DNS load balancing with weighted-round-robin • Supports hostnames for non-AWS infrastructure

Route53 and MongoDB • Short answer: use hostnames for all components • With replica sets, hostnames can ease machine replacement • With sharded clusters, hostnames can simplify config server maintenance

VPC • Virtual Private Cloud lets you provision a logically isolated network insideAWS • You manage all aspects of networking including – IP address ranges – Subnets – Routing tables and gateways • Can be used as an extension to an offsite data center with Hardware VPN

VPC Public and Private http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_In troduction.html • Private subnets hidden to outside world • Internet Gateway and EIPs can be used to access • Web tier in public subnet • Data tier in private subnet

ElastiCache • Distributed in-memory cache • Backed by Memecached or Redis • Can be a drop-in replacement for existing cache deployments • Supports auto-discovery and read-replicas

Future Directions

RedShift • Fully-managed petabyte-scale data warehouse service • MongoDB not natively supported as a data source • … So how do you get your data in?

Data Pipeline • Process and move data between differentAWS compute and storage services • Date Pipeline handles resources, failures, and dependencies http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/what-is-datapipeline.html

Data Pipeline with MongoDB AWS Data Pipeline MongoDB S3 EMRor Redshift

OpsWorks • Complete DevOps stack • Model and manage apps, load balancers, databases • Uses Chef recipes • Load or time-based scaling • Deploying MongoDB with OpsWorks: – http://blogs.aws.amazon.com/application- management/post/Tx1RB65XDMNVLUA/Deploying- MongoDB-with-OpsWorks

CloudWatch • Monitoring for AWS resources • Supports custom metrics http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatch.ht ml

aws cloudwatch put-metric-data --metric-name ResidentMemory --namespace MongoDB --timestamp 2014-02-14T20:30:00Z --value 32 --unit Gigabytes CloudWatch Custom Metrics

Questions?

MongoDB World New York City, June 23-25 #MongoDBWorld See what’s next in MongoDB including • MongoDB 2.6 • Sharding • Replication • Aggregation http://world.mongodb.com Save 25% with discount code 25SandeepParikh

http://www.mongodb.com	582
https://www.mongodb.com	34
http://www.slideee.com	12
https://live.mongodb.com	4

MongoDB and AWS: Integrations

by MongoDB

on Apr 24, 2014

Statistics

Views

Actions

4 Embeds 632

Accessibility

Categories

Upload Details

Usage Rights

Report content

MongoDB and AWS: Integrations Presentation Transcript