Tricks And Tradeoffs Of Deploying My Sql Clusters In The Cloud

MySQL & Amazon EC2 @RightScale • Operating in Amazon EC2 since fall 2006 • Cloud Computing Management System • Replicated MySQL product offering mid ‘07 – 100’s of customer deployments – Automation to launch slaves, failover, backup, … • Animoto.com scaled to 4000 servers in 3.5 days © 2009 RightScale Inc — All rights reserved 2

Cloud Computing & Databases As many DB machines as 1. Infinite computing resources you want! 2. Available on-demand No cost for 3. Pay per use short periods 4. Delivered across the Internet Save work, More reliable v Fully Automatable Infrastructure © 2009 RightScale Inc — All rights reserved 3

Cloud Computing Vision • No up-front costs • Easy Dev & Test load clone time mon tue wed thu fri sat sun • Scale batch apps • Pay for average, not peak load peak load average load time mon tue wed thu fri sat sun © 2009 RightScale Inc — All rights reserved 4

Scale-up/down Automation • Scale-up www.mysite.com App server load > 50% – Majority voting – Apache Apache Add app servers – HAproxy HAproxy Tie into load bal & db – • Scale-down – App server load < 20% App Srvr App Srvr App Srvr App Srvr N/array 1/array – Majority voting MySQL MySQL replication master slave © 2009 RightScale Inc — All rights reserved 5

Starcut (Olympics) § Low cost to entry, scale up as demand grows § Repeatable, ”clean-room” design promotes stability and process § Great sandbox for testing and benchmarking © 2009 RightScale Inc — All rights reserved 8

Starcut (Stocks) Daily scale up/down to save costs § Global reach for services is needed (Europe, USA, Asia) § Big name clients demand carrier-grade reliability § Full 24/7, 365 support expectations to us and us to § our vendors © 2009 RightScale Inc — All rights reserved 9

Amazon EC2 Storage Architecture ½ core … 8 cores $0.10-$0.80/hr CPU 1.7GB … 16GB Memory 1 x 160GB … 4 x 420GB Local disks 1GB … 1TB volumes $0.10/GB/mo 1 … volumes / server Scalable $0.10/M-io-ops EBS disk array network attached, redundant redundant back-end store $0.15/GB/mo Persistent S3 Store S3 put/get semantics $0.10/K-ops © 2009 RightScale Inc — All rights reserved 11

Elastic Block Store (EBS) • Create volume: 1GB – 1TB size • Attach to any one instance in same availability zone • Pay $0.10/GB per month + $0.10/million I/O ops • Snapshot backup a volume to S3 • Create new volume from snapshot © 2009 RightScale Inc — All rights reserved 12

Snapshots Server 2 Server 3 Server 4 Server 1 • Point-in-time Vol 2 Vol 3 Vol 4 Vol 1 snapshot • Written to S3 – (Not to bucket!) snap 2 snap 1 Amazon S3 • Incremental snapshots • Restore to new volume – Instantaneous (lazy restore) © 2009 RightScale Inc — All rights reserved 13

Keeping Safe • Freeze data while snapshotting – XFS filesystem has freeze command – But: 2.6.18 kernel bug causes hung filesystem (ouch!) • Volumes are fragile in transit! – Unmounting can leave data in flight – Mounting mistake will be destructive • Use snapshots for safety! – Always snapshot after unmount – Create fresh volume from snapshot, mount that, retry in case of problem © 2009 RightScale Inc — All rights reserved 15

MySQL/EBS Replication App App Server 1 Server 2 MySQL MySQL replication master slave EBS EBS vol vol daily 10-minute snapshots snapshots Amazon Amazon S3 S3 © 2009 RightScale Inc — All rights reserved 16

Database operations pain points • Master hardware failure no replica ‘til hw fixed • Master software failure no replica ‘til root caused • Testing schema change extra system: 99% idle • Migrate to new hardware 2x rack space, SAN ports, … • New DB requirement machines, ports, disks, … • QA, test, stage, DR, demo, … always to few, always too idle © 2009 RightScale Inc — All rights reserved 17

Failure? Allocate fresh resources! MySQL MySQL MySQL MySQL MySQL dead replication master replication slave replication next replication next EBS EBS EBS EBS EBS vol vol vol vol vol snapshot snapshot snapshot snapshot snapshot init init Amazon Amazon Amazon Amazon Amazon S3 S3 S3 S3 S3 time • Extra resources are free for short periods! (e.g. 4-core/16GB + 1TB EBS => <$30/day © 2009 RightScale Inc — All rights reserved 18

Vertical Database Scaling Clone Server Config 1 www.rightscaledemo.com Load bal Load bal App Srvr App Srvr Small Small replication master slave clone config Large slave © 2009 RightScale Inc — All rights reserved 19

Vertical Database Scaling Clone Server Config 1 www.rightscaledemo.com Launch large slave 2 Load bal Load bal App Srvr App Srvr Small Small replication master slave replication Launch Large slave © 2009 RightScale Inc — All rights reserved 20

Vertical Database Scaling Clone Server 1 www.rightscaledemo.com Launch large slave 2 Load bal Load bal Promote to master 3 App Srvr App Srvr Small Small replication master slave replication master-db.rightscaledemo.com Large slave © 2009 RightScale Inc — All rights reserved 21

Vertical Database Scaling Clone Server 1 www.rightscaledemo.com Launch large slave 2 Load bal Load bal Promote to master 3 App Srvr App Srvr Launch new slave 4 Watch demo in Small Small replication master slave webinar recording at replication www.rightscale.com/webinars Launch Large Large replication master slave © 2009 RightScale Inc — All rights reserved 22

Cloud Principle #1: grab the next one! • Database management – Always roll forward, never fail-back – Animoto upgraded L -> XL @600 servers • App server management – Always roll forward, never fix in-place – www1 bust? Launch www3! Then troubleshoot • Good sysadmins have the urge to fix… L © 2009 RightScale Inc — All rights reserved 23

Many small databases Database pain cycle: 1. Recognize the DB is important 2. Buy expensive DB hardware 3. Consolidate DBs onto expensive hardware 4. Recognize the DB is now even more critical 5. Buy more expensive DB hardware 6. Repeat! Cloud Computing = Infinite servers and disks + Fully automatable © 2009 RightScale Inc — All rights reserved 25

Many Small Databases • Sharding – Split by user – Need to manage mappings • Read slaves – Many slave to increase read performance – Need to manage failover © 2009 RightScale Inc — All rights reserved 26

EBS Performance • Functionality (snapshots) trump performance • Very difficult to benchmark – Multiple levels of cache, some hidden • Sequential I/O limited by network – 70-90MB/sec typ, 100MB/sec max – 4 local drives striped yield more • Random I/O (sustained) – 400-1000 read ops/sec – 100-400 write ops/sec • Stripe to increase performance © 2009 RightScale Inc — All rights reserved 27

Clouds are Multiplying! – Amazon US + EU – GoGrid, Flexiscale – Eucalyptus open source release – RackSpace CloudServers – Sun cloud announcement – And many more… • Operating principles are the same – Storage architectures differ – From better local drives to traditional SANs © 2009 RightScale Inc — All rights reserved 28

Public vs. Private Cloud • Public cloud: Cloud Definition: – shared cloud computing 1. Infinite computing resources 2. Available on-demand infrastructure 3. Pay per use – connected to the public 4. Delivered across the Internet internet • Private cloud: – cloud infrastructure dedicated to a single party – may not be connected to the public internet • Hybrid cloud – private and public clouds used together – leverage the benefits of both © 2009 RightScale Inc — All rights reserved 29

Hybrid Cloud Use-Cases • Develop in public, deploy in private – Outsourced/distributed development – flexibility of test & dev resources • Develop in private, deploy in public – existing in house dev resources – ability to scale in public cloud – public-facing app • Private core, public expansion – Transaction intensive in-house core – Batch processing in public cloud © 2009 RightScale Inc — All rights reserved 30

RightScale Supports Hybrid Clouds 1. Commandeer some machines 2. Install Eucalyptus 3. Register with RightScale 4. The power of RightScale for your cloud 5. Invite friends to your cloud © 2009 RightScale Inc — All rights reserved 31

Clouds Surpass Traditional Hosting Servers on-demand • Disk Volumes on-demand • Multiple datacenters • Reassign IPs / VIPs across datacenters • Multiple geographies • Multiple providers • • Automation tying it all together © 2009 RightScale Inc — All rights reserved 33

Tricks And Tradeoffs Of Deploying My Sql Clusters In The Cloud

by MySQLConference on May 11, 2009

Statistics

Views

Actions

0 Embeds 0

Accessibility

Categories

Upload Details

Usage Rights

Report content

Tricks And Tradeoffs Of Deploying My Sql Clusters In The Cloud Presentation Transcript