{"id":2302,"date":"2019-01-04T18:49:38","date_gmt":"2019-01-04T18:49:38","guid":{"rendered":"https:\/\/www.danielpradilla.info\/blog\/?p=2302"},"modified":"2019-01-04T21:35:50","modified_gmt":"2019-01-04T21:35:50","slug":"aws-lambda-python","status":"publish","type":"post","link":"https:\/\/www.danielpradilla.info\/blog\/aws-lambda-python\/","title":{"rendered":"10 things I learned while deploying my first python function to AWS Lambda"},"content":{"rendered":"

\"\"<\/a><\/p>\n

I spent a few days on and off trying to deploy a Flask<\/a> REST service to AWS Lambda<\/a>, just to experience what the cool kids were talking about. These are some of the things I learned along the way:<\/p>\n

 <\/p>\n

Zappa is the easiest packager\/deployer for python (as of December 2018)<\/h3>\n

Zappa<\/a> provides good quality feedback on the packaging\/deployment process. It’s compatible with all the popular python REST frameworks. Minimally configure, deploy and you are done. It packages your whole environment and your own modules, so you will face minimal “module not found” errors.<\/p>\n

Zappa provides a “tail” function that allows you to debug the errors in your deployment directly from the command line.<\/p>\n

 <\/p>\n

Zappa doesn’t work properly with Anaconda (as of December 2018)<\/h3>\n

I use Anaconda for environment management. Zappa is geared towards venv users. There are “hacky” ways of making it work. Setting the VIRTUAL_ENV environment variable, according to https:\/\/github.com\/Miserlou\/Zappa\/issues\/167<\/a>, got me a long way. But I had a third-party module that kept failing. I spent 6 hours of a weekend that I’ll never get back on this.<\/p>\n

 <\/p>\n

Anaconda’s environment management is very different than venv<\/h3>\n

Files are stored elsewhere (conda info –envs is your friend). I’ve used Anaconda since forever, so I didn’t know that venv stored files locally within your project. Ugh! (am I getting this wrong? please tell me so). By default, Anaconda stores your requirements outside of your project folder. Naturally, this wreaks havoc with anything that is expecting an environment folder within your project.<\/p>\n

 <\/p>\n

Chalice is almost as good as Zappa<\/h3>\n

Chalice<\/a> is the native tool from AWS for doing these kind of things. It’s not a packager \/ deployer like Zappa but a whole framework, so you have to refactor your code from whatever framework you are using. Fortunately for me the syntax is almost the same as Flask.<\/p>\n

Chalice relies on your requirements.txt as a guide to package the dependencies of your lambda functions. Good if you have a messy environment.<\/p>\n

Debugging the deployment is brutal.<\/p>\n

 <\/p>\n

Chalice doesn’t automatically package your own modules<\/h3>\n

According to the documentation<\/a> you have to put all your modules in a magical “chalicelibs” directory. Even after doing that, I still had import problems (importing a local module that was importing another local module). I solved it by spelling out the location, using “from . import mymodule”<\/p>\n

 <\/p>\n

AWS Lambda packages are restricted to 50MB<\/h3>\n

Messy environment? too many requirements? you are out of luck. AWS Lambda packages are limited to 50MB, zipped<\/a>.<\/p>\n

I actually had a huge 28MB library (zipped!) that I had to strategically trim down (hi, technical debt!) in order to fit it into my package.<\/p>\n

Probably this is a sign that I shouldn’t be uploading a python class, but the actual methods independently. I know, it’s lambda functions<\/strong> not lambda class with everything and the kitchen sink<\/strong><\/p>\n

 <\/p>\n

You can upload a class with a bunch of methods and a REST api and enjoy the benefits of serverless<\/h3>\n

Not the best pattern, not the most efficient solution, but hey, for small stuff works. You get a million requests for free.<\/p>\n

 <\/p>\n

You can decrease the response time if you increase the memory allocation of your function<\/h3>\n

I was getting a 500ms response time, good but an order of magnitude slower that I was getting on my laptop. Until I read this text below the “Memory” slider in the Lambda console<\/p>\n

“Your function is allocated CPU proportional to the memory configured.”<\/p>\n<\/blockquote>\n

I moved the slider to 1024MB and the response time went down to 125ms!<\/p>\n

 <\/p>\n

Lambda functions are a good solution for APIs<\/h3>\n

I’ve been using Docker since 2016 for packaging small python APIs, but deploying them and managing the Dockerfiles is kind of a pain. If you manage to refactor your application down to single-purpose methods, AWS Lambda offers an impressively easy way to deploy pay-per-use load-balanced functions in a secure server whose hardware and OS stack you don’t have to manage. This schema works best for functions that will be run sporadically and don’t need an always-on server.
\nCome to think of it, few functions are rarely running *all* the time, and if the inputs are the same, then you can leverage the Cloudfront cache. This cost calculator<\/a> might help you estimate total cost once you deploy.<\/p>\n

 <\/p>\n

Links that saved me (beside the docs)<\/h3>\n

The Right Way™ to do Serverless in Python<\/a><\/p>\n

Building Serverless Python Apps Using AWS Chalice<\/a><\/p>\n

The fear and frustration of migrating a simple web app to serverless<\/a><\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

I spent a few days on and off trying to deploy a Flask REST service to AWS Lambda, just to experience what the cool kids were talking about. These are some of the things I learned along the way:   Zappa is the easiest packager\/deployer for python (as of December 2018) Zappa provides good quality… Continue reading 10 things I learned while deploying my first python function to AWS Lambda<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[331],"tags":[],"class_list":["post-2302","post","type-post","status-publish","format-standard","hentry","category-software-development-en-en","entry"],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1tlzy-B8","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2399,"url":"https:\/\/www.danielpradilla.info\/blog\/my-python-logging-setup\/","url_meta":{"origin":2302,"position":0},"title":"My Python logging setup","author":"Daniel Pradilla","date":"24\/08\/2023","format":false,"excerpt":"Working on various Python projects has taught me the importance of consistent logging, especially when dealing with distributed computing frameworks like Spark. Logging is not just about keeping track of errors or information; it's about having a detailed and systematic record of the operations to understand the flow of your\u2026","rel":"","context":"In "Software Dev."","block_context":{"text":"Software Dev.","link":"https:\/\/www.danielpradilla.info\/blog\/category\/software-development-en-en\/"},"img":{"alt_text":"My Python logging setup","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2023\/08\/5aadc010-b913-4eb0-9388-cd8d6c743271.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2023\/08\/5aadc010-b913-4eb0-9388-cd8d6c743271.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2023\/08\/5aadc010-b913-4eb0-9388-cd8d6c743271.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2023\/08\/5aadc010-b913-4eb0-9388-cd8d6c743271.jpg?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":2229,"url":"https:\/\/www.danielpradilla.info\/blog\/linear-optimization-with-or-tools-containerizing-a-gunicorn-web-application\/","url_meta":{"origin":2302,"position":1},"title":"Linear optimization with or-tools: containerizing a gunicorn web application","author":"Daniel Pradilla","date":"15\/05\/2018","format":false,"excerpt":"Previously, we left our app working with our local python+gunicorn+nginx installation. In order to get there we had to do quite a bit of configuration and if we wanted to deploy this in a server or send it to a friend, we would have to go through a very error-prone\u2026","rel":"","context":"In "Software Dev."","block_context":{"text":"Software Dev.","link":"https:\/\/www.danielpradilla.info\/blog\/category\/software-development-en-en\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2018\/05\/docker.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2018\/05\/docker.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2018\/05\/docker.jpg?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":2205,"url":"https:\/\/www.danielpradilla.info\/blog\/linear-optimization-with-or-tools\/","url_meta":{"origin":2302,"position":2},"title":"Linear Optimization with or-tools","author":"Daniel Pradilla","date":"07\/06\/2017","format":false,"excerpt":"\u00a0 Getting started Over the last couple of months I've been getting my feet wet with linear programming and mathematical optimisation. I got a sense of how it all worked from this Discrete Optimisation course in Coursera and googling around I discovered that there are a ton of tools out\u2026","rel":"","context":"In "Best of"","block_context":{"text":"Best of","link":"https:\/\/www.danielpradilla.info\/blog\/category\/bestof\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/06\/grocery_bag_brown_bag.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/06\/grocery_bag_brown_bag.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/06\/grocery_bag_brown_bag.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/06\/grocery_bag_brown_bag.jpg?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":2212,"url":"https:\/\/www.danielpradilla.info\/blog\/linear-optimization-with-or-tools-building-a-web-front-end-with-falcon-and-gunicorn\/","url_meta":{"origin":2302,"position":3},"title":"Linear Optimization with or-tools \u00e2\u20ac\u201d building a web front-end with falcon and gunicorn","author":"Daniel Pradilla","date":"14\/11\/2017","format":false,"excerpt":"In a previous post, I put together a script for solving a linear optimisation problem using Google's OR-tools. This python script is callable from the command line and you kinda need to know what you are doing and how to organize the parameters. So, in order to address this difficulty,\u2026","rel":"","context":"In "Software Dev."","block_context":{"text":"Software Dev.","link":"https:\/\/www.danielpradilla.info\/blog\/category\/software-development-en-en\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/11\/groceryshopping.gif?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/11\/groceryshopping.gif?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/11\/groceryshopping.gif?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/11\/groceryshopping.gif?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":2245,"url":"https:\/\/www.danielpradilla.info\/blog\/recommender-system-for-finding-subject-matter-experts-using-the-enron-email-corpus\/","url_meta":{"origin":2302,"position":4},"title":"Recommender system for finding subject matter experts using the Enron email corpus","author":"Daniel Pradilla","date":"20\/11\/2018","format":false,"excerpt":"This is a little project to create a recommender system to find mentors inside an organization, using Natural Language Processing. It started as an excuse to build a data visualization I had in mind: an interactive word cloud that did something. When I started, I didn't know anything about Topic\u2026","rel":"","context":"In "Best of"","block_context":{"text":"Best of","link":"https:\/\/www.danielpradilla.info\/blog\/category\/bestof\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2018\/11\/bubbles2.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2018\/11\/bubbles2.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2018\/11\/bubbles2.jpg?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":2300,"url":"https:\/\/www.danielpradilla.info\/blog\/uploading-and-downloading-documents-from-amazon-s3-using-bash\/","url_meta":{"origin":2302,"position":5},"title":"Uploading and downloading documents from Amazon S3 using bash","author":"Daniel Pradilla","date":"12\/12\/2018","format":false,"excerpt":"You need to upload a file to S3 and cannot install new packages in the server, nor the s3 client tools. You only have bash, openssl and sed. Go. I found and adapted a script by\u00a0Viktor Szakats, that creates all the proper headers expected by the latest AWS API. I\u2026","rel":"","context":"In "Software Dev."","block_context":{"text":"Software Dev.","link":"https:\/\/www.danielpradilla.info\/blog\/category\/software-development-en-en\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/posts\/2302","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/comments?post=2302"}],"version-history":[{"count":0,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/posts\/2302\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/media?parent=2302"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/categories?post=2302"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/tags?post=2302"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}