(SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

Transcript of "(SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014"

1. 1. SDD424 Simplifying Scalable Distributed Applications Using DynamoDB Streams
2. 2. •WhyDynamoDBStreams •What DynamoDBStreams •How Demo @akshatvig
3. 3. use cases Online gaming Ad tech Livevoting Social media Mobile messaging Backup & restore @akshatvig
4. 4. replication @akshatvig
5. 5. backups @akshatvig
6. 6. views @akshatvig
7. 7. Reactors –the new triggers @akshatvig
8. 8. Logs @akshatvig
9. 9. Op: PUT John Tokyo Op: UPDATE John Pluto Op: UPDATE John Mars Logsand databases Name Destination Tokyo Mars @akshatvig
10. 10. istruthconstruct state @akshatvig
11. 11. Logs arepowerful. Atomicity, consistency, and durability Replication Point-in-time restores Materialized views Auditing And much more... @akshatvig
12. 12. What isDynamoDBStreams? It is a stream of updates Scales with your table DynamoDBStreams DynamoDB @akshatvig
13. 13. Social networkapplication Comments and notifications @akshatvig
14. 14. Take 1 Comments Table @akshatvig
15. 15. post Take 2 Comments Table Processor @akshatvig
16. 16. Take3 Comments Table @akshatvig
17. 17. Globalusers @akshatvig
18. 18. Cross region library @akshatvig
19. 19. groups @akshatvig
20. 20. Cross region replication post @akshatvig
21. 21. Cross region library Horizontal scaling : Workers Load balancing @akshatvig
22. 22. Cross region library Fault tolerant Checkpointing @akshatvig
23. 23. Cross region replication Shard 1 Partition 1 Shard 2 Shard 3 Partition 2 KCL Worker KCL Worker KCL Worker @akshatvig
24. 24. Cross region replication Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Shard 1 Shard 2 Shard 3 Shard 4 KCL Worker KCL Worker KCL Worker KCL Worker @akshatvig
25. 25. Cross region replication library @akshatvig
26. 26. Cross region replication library @akshatvig
27. 27. DynamoDB Streams and DynamoDB Connectors simplify cross region replication! @akshatvig
28. 28. Materialized view Extending DynamoDBcapabilities @akshatvig
29. 29. Cross region Streams Streams Cross region @akshatvig
30. 30. Data Tier Amazon DynamoDB Amazon RDS Amazon ElastiCache Amazon S3 Amazon CloudSearch Amazon Redshift archive rich search read replicas hot reads analytics complex queries & transactions purpose @akshatvig
31. 31. Features of @akshatvig
32. 32. View Type Destination Old Image –Before update Name= John, Destination = Mars New Image –After update Name = John,Destination = Pluto Old and New Images Name = John,Destination = Mars Name = John, Destination = Pluto Keys Only Name = Johntypes @akshatvig
33. 33. Asynchronous i=A i=A ack @akshatvig
34. 34. Exactlyonce i=A i=B i=B @akshatvig
35. 35. Strictly ordered records i=A i=B i=C i=C @akshatvig
36. 36. Durability & high availability High throughput consensus protocol Replicated across multiple AZs @akshatvig
37. 37. Managed @akshatvig
38. 38. Elasticity Adjusts @akshatvig
39. 39. Performance Sub-second latency @akshatvig
40. 40. Retention period Records available for 24 hours @akshatvig
41. 41. DynamoDB Local Desktopinstallable Development & testing Publiclyavailable -http://bit.ly/1yt1r9q Now supports DynamoDB Streams @akshatvig
42. 42. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start(); @akshatvig
43. 43. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start(); @akshatvig
44. 44. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start(); @akshatvig
45. 45. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start(); @akshatvig
46. 46. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ... @akshatvig
47. 47. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ... @akshatvig
48. 48. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ... @akshatvig
49. 49. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ... @akshatvig
50. 50. Cross region replication @akshatvig
51. 51. DynamoDB Reactors Trigger Lambda functions Example –Validate address, send notifications Console support @akshatvig
52. 52. Preview available Temporary endpoints: N. Virginia & Ireland Available until global launch Registerfor preview at http://amzn.to/11dh9M0 @akshatvig
53. 53. Preview console TBD: Console snapshot
54. 54. @akshatvig
55. 55. Landing pagehttp://amzn.to/1xodVzQ @akshatvig
56. 56. Please give us your feedback on this session. Complete session evaluations and earn re:Invent swag. http://bit.ly/awsevals