How to build scalable, resilient applications in AWS Cloud using Simple Workflow Service. Cristian Balcanu

Transcription

1 How to build scalable, resilient applications in AWS Cloud using Simple Workflow Service Cristian Balcanu

2 Agenda What is SWF How to use it The problems solved by SWF Questions

3 What is Simple Workflow Service SWF

4 Workflow? A workflow consists of a sequence of concatenated (connected) steps. Wikipedia definition

5 Who is using SWF?

6

7 Mars "Curiosity Rover"

8 Jet Propulsion Laboratory (aka JPL) NASA

9 For what exactly does JPL use SWF?

10 For what exactly does JPL use SWF?

11

12

13

14 SWF Actors

15 Amazon SWF Workflow History Decider Workers Activity Workers

16 How to write code that uses SWF?

17

18 SWF API Raw SWF API 31 methods

19 Flow Framework

20 How to Declare an public interface ImageProcessingActivities String processimage(string img); String generatestereodetail(string leftimg, String rightimg);

21 How to Implement an Activity public class ImageProcessingActivitiesImpl implements ImageProcessingActivities public String processimage(string img) { // regular java code here. public String generatestereodetail(string leftimg, String rightimg) { // regular java code here. } }

22 How to Declare CloseTimeoutSeconds = 60) public interface StereoWorkflow { = "1.0") public void generatestereoimage( String leftimg, String rightimg);

23 How to Implement a Workflow public class StereoWorkflowImpl implements StereoWorkflow ImageProcessingActivitiesClient ImageTilingWorkflowClient tilingworkflow; Left Image Right Image Warp \ / Stereo Match 3D Map Tiling public void generatestereoimage(string leftimg, String rightimg) { Promise<String> processedleftimg = imageprocessor.processimage(leftimg); Promise<String> processedrightimg = imageprocessor.processimage(rightimg); imageprocessor.generatestereodetail(processedleftimg, processedrightimg); } } tilingworkflow.tileimage(processedleftimg); tilingworkflow.tileimage(processedrightimg);

24 How are things working together?

25 Amazon SWF Image Processing Workflow History Decider Workers Activity Workers

26 Workflow Execution History 1. Workflow started 2. A decision must be made

27 Amazon SWF Decision Image Processing Workflow History Decider Workers Activity Workers

28 public class StereoWorkflowImpl implements StereoWorkflow ImageProcessingActivitiesClient ImageTilingWorkflowClient tilingworkflow; public void generatestereoimage(string leftimg, String rightimg) { Promise<String> processedleftimg = imageprocessor.processimage(leftimg); > Promise<String> processedrightimg = imageprocessor.processimage(rightimg); imageprocessor.generatestereodetail(processedleftimg, processedrightimg); } } tilingworkflow.tileimage(processedleftimg); tilingworkflow.tileimage(processedrightimg);

29 Workflow Execution History 1. Workflow started 2. A decision must be made 3. A decision was made: schedule activities "processimage(leftimg)" and "processimage(rightimg)"

30 Amazon SWF processimage processimage (rightimg) (leftimg) Image Processing Workflow History Decider Workers Activity Workers

31 Workflow Execution History 1. Workflow started 2. A decision must be made 3. A decision was made: schedule activities "processimage(leftimg)" and "processimage(rightimg)" 4. Activity "processimage(leftimg)" was polled by host1 5. Activity "processimage(rightimg)" was polled by host2

32 Amazon SWF Image Processing Workflow History processimage (rightimg) Decider Workers Activity Workers

33 Workflow Execution History 3. A decision was made: schedule activities "processimage(leftimg)" and "processimage(rightimg)" 4. Activity "processimage(leftimg)" was polled by host1 5. Activity "processimage(rightimg)" was polled by host2 6. Activity "processimage(rightimg)" started at step 5 was executed successfully with result X 7. A decision must be made

34 Amazon SWF Decision Image Processing Workflow History Decider Workers Activity Workers

35 public class StereoWorkflowImpl implements StereoWorkflow ImageProcessingActivitiesClient ImageTilingWorkflowClient public void generatestereoimage(string leftimg, String rightimg) { Promise<String> processedleftimg = imageprocessor.processimage(leftimg); Promise<String> processedrightimg = imageprocessor.processimage(rightimg); imageprocessor.generatestereodetail(processedleftimg, processedrightimg); > } } tilingworkflow.tileimage(processedleftimg); tilingworkflow.tileimage(processedrightimg);

36 Workflow Execution History 4. Activity "processimage(leftimg)" was polled by host1 5. Activity "processimage(rightimg)" was polled by host2 6. Activity "processimage(rightimg)" started at step 5 was executed successfully with result X 7. A decision must be made 8. A decision was made: signal tiling workflow "tileimage(processedrightimage)"

37 Tile Workflow Decider Workers Tile Workflow History Amazon SWF Image Processing Workflow History tileimage(processedrightimage) Decider Workers Activity Workers

38 Amazon SWF Image Processing Workflow History processimage (leftimg) Decider Workers Activity Workers

39 Workflow Execution History 6. Activity "processimage(rightimg)" started at step 5 was executed successfully with result X 7. A decision must be made 8. A decision was made: signal tiling workflow "tileimage(processedrightimage) 9. Activity started at step 4 was executed successfully with result Y 10. A decision must be made

40 Amazon SWF Decision Image Processing Workflow History Decider Workers Activity Workers

41 public class StereoWorkflowImpl implements StereoWorkflow ImageProcessingActivitiesClient ImageTilingWorkflowClient public void generatestereoimage(string leftimg, String rightimg) { Promise<String> processedleftimg = imageprocessor.processimage(leftimg); Promise<String> processedrightimg = imageprocessor.processimage(rightimg); > imageprocessor.generatestereodetail(processedleftimg, processedrightimg); > } } tilingworkflow.tileimage(processedleftimg); tilingworkflow.tileimage(processedrightimg);

42 Workflow Execution History 8. A decision was made: signal tiling workflow "tileimage(processedrightimage) 9. Activity started at step 4 was executed successfully with result Y 10. A decision must be made 11. A decision was made: schedule activity generatestereodetail and signal tiling workflow "tileimage(procesedleftimage)"

43 Tile Workflow Decider Workers Tile Workflow History Amazon SWF Image Processing Workflow tileimage(processedlefthistory generatestereodetail Image) Decider Workers Activity Workers

44 Workflow Execution History 9. Activity started at step 4 was executed successfully with result Y 10. A decision must be made 11. A decision was made: schedule activity generatestereodetail and signal tiling workflow "tileimage(procesedleftimage)" 12. Activity generatestereodetail was polled by host3

45 generatestereode Amazon SWF Image Processing Workflow History Decider Workers Activity Workers

46 Workflow Execution History 11. A decision was made: schedule activity generatestereodetail and signal tiling workflow "tileimage(procesedleftimage)" 12. Activity generatestereodetail was polled by host3 13. Activity started at step 12 was executed successfully 14. A decision must be made

47 public class StereoWorkflowImpl implements StereoWorkflow ImageProcessingActivitiesClient ImageTilingWorkflowClient public void generatestereoimage(string leftimg, String rightimg) { Promise<String> processedleftimg = imageprocessor.processimage(leftimg); Promise<String> processedrightimg = imageprocessor.processimage(rightimg); imageprocessor.generatestereodetail(processedleftimg, processedrightimg); } } tilingworkflow.tileimage(processedleftimg); tilingworkflow.tileimage(processedrightimg);

48 Workflow Execution History 11. A decision was made: schedule activity generatestereodetail and signal tiling workflow "tileimage(procesedleftimage)" 12. Activity generatestereodetail was polled by host3 13. Activity started at step 12 was executed successfully 14. A decision must be made 15. A decision was made: workflow finished with success

49 Amazon SWF Image Processing Workflow History (closed) Image Processing Workflow History Decider Workers Activity Workers

50 Key SWF Features

51 Fault Tolerance

52 Scalability and Efficiency

53 Amazon SWF Workflow History Decider Workers Activity Workers

54 Visibility and Control

55 Failures

56 Failure Detection in SWF

57 Decision Timeout Start-to-Finish Schedule Time Poll Time Decisions are reported to SWF Decision is executed Time

58 Activity Timeouts Schedule-to-Start Schedule Time Poll Time Result is reported to SWF Activity is executed Time

59 Activity Timeouts Start-to-Finish Schedule Time Poll Time Result is reported to SWF Activity is executed Time

60 Activity Timeouts Schedule-to-Finish Schedule Time Poll Time Result is reported to SWF Activity is executed Time

61 Activity Timeouts Heartbeat Timeout Schedule Time Poll Time Result is reported to SWF Activity is executed Time

62 How to handle failures?

63 Retry the call

64 @Activities public interface ImageProcessingActivities = 5, retryexpirationseconds = 600, maximumretryintervalseconds = String processimage(string = 5, retryexpirationseconds = 600, maximumretryintervalseconds = String generatestereodetail(string leftimg, String rightimg); }

65 Conclusion AWS SWF is a tool that makes orchestration available in the cloud.

66 Conclusion

67 Resources

68 Questions?