{ "cells": [ { "cell_type": "markdown", "source": [ "# Example App - Sync Two S3 Folder\n", "\n", "In this tutorial, you will learn how to create a simple app that can sync files from one folder to another folder in S3 while preserving the same folder structure in real-time.\n", "\n", "To get started, you should first [configure an S3 put object event trigger](https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html) that can monitor any changes made to the source S3 folder. After that, you can proceed to [create a Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html) that can handle the event and sync the file to the target S3 folder." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 4, "outputs": [], "source": [ "# this is your lambda function code\n", "\n", "from s3pathlib import S3Path\n", "\n", "s3dir_source = S3Path(\"s3pathlib/example-app/sync-two-s3-folders/source/\")\n", "s3dir_target = S3Path(\"s3pathlib/example-app/sync-two-s3-folders/target/\")\n", "\n", "def lambda_handler(event, context):\n", " # parse s3 put object event\n", " s3path_source = S3Path(\n", " event[\"Records\"][0][\"s3\"][\"bucket\"][\"name\"],\n", " event[\"Records\"][0][\"s3\"][\"object\"][\"key\"],\n", " )\n", " # find out the target s3 location\n", " s3path_target = s3dir_target.joinpath(\n", " s3path_source.relative_to(s3dir_source)\n", " )\n", " # copy data\n", " s3path_source.copy_to(s3path_target, overwrite=True)" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "code", "execution_count": 13, "outputs": [ { "data": { "text/plain": "S3Path('s3://s3pathlib/example-app/sync-two-s3-folders/target/')" }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# firstly, we clean up the source and target location\n", "s3dir_source.delete()\n", "s3dir_target.delete()" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "code", "execution_count": 14, "outputs": [ { "data": { "text/plain": "S3Path('s3://s3pathlib/example-app/sync-two-s3-folders/source/folder/file.txt')" }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# then we create a file in the source location\n", "s3path_source = s3dir_source.joinpath(\"folder/file.txt\")\n", "s3path_source.write_text(\"hello\")" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "code", "execution_count": 15, "outputs": [ { "data": { "text/plain": "0" }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# at begin, the s3 target folder doesn't have any file\n", "s3dir_target.count_objects()" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "code", "execution_count": 16, "outputs": [], "source": [ "# we use this code to simulate the s3 put object event\n", "event = {\n", " \"Records\": [\n", " {\n", " \"s3\": {\n", " \"bucket\": {\"name\": s3path_source.bucket},\n", " \"object\": {\"key\": s3path_source.key}\n", " }\n", " }\n", " ]\n", "}\n", "lambda_handler(event, None)" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "code", "execution_count": 11, "outputs": [ { "data": { "text/plain": "[S3Path('s3://s3pathlib/example-app/sync-two-s3-folders/target/folder/file.txt')]" }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# it should have one file now\n", "s3dir_target.iter_objects().all()" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 0 }