Function Name:<\/strong> to trace back the exact point of failure or point of interest.<\/p>\n <\/p>\n
My Python Logging Setup<\/h3>\n
Here\u2019s a basic setup I use to ensure both the timestamp and the function name are always logged:<\/p>\n
\r\nimport logging\r\n\r\nlogging.basicConfig(\r\n level=logging.INFO, \r\n format='%(asctime)s [%(levelname)s] - %(funcName)s: %(message)s',\r\n datefmt='%Y-%m-%d %H:%M:%S'\r\n)\r\n\r\nlogger = logging.getLogger(__name__)\r\n\r\ndef sample_function():\r\n logger.info("This is a sample log entry.")\r\n\r\n\r\n<\/pre>\nThe above code will produce an output like this:<\/p>\n
2023-08-19 14:30:25 [INFO] - sample_function: This is a sample log entry.<\/pre>\nDifferent stages of development or different scenarios might require varying levels of log detail. Sometimes, you want to capture every single event (debug mode), other times just informational messages, and on certain occasions, only errors. Having the ability to dynamically adjust the log level provides this flexibility.<\/p>\n
\r\n\r\ndef setup_logger(level='INFO'):\r\n numeric_level = getattr(logging, level.upper(), None)\r\n if not isinstance(numeric_level, int):\r\n raise ValueError(f'Invalid log level: {level}')\r\n\r\n logging.basicConfig(\r\n level=numeric_level, \r\n format='%(asctime)s [%(levelname)s] - %(funcName)s: %(message)s',\r\n datefmt='%Y-%m-%d %H:%M:%S'\r\n )\r\n return logging.getLogger(__name__)\r\n\r\n# Example usage:\r\nlogger = setup_logger('DEBUG')\r\n\r\ndef sample_function():\r\n logger.debug("This is a debug message.")\r\n logger.info("This is an info message.")\r\n logger.error("This is an error message.")\r\n\r\n<\/pre>\nWhenever I use AWS Glue, I can configure a parameter called LOG_LEVEL which I can set to the desired value depending on the development stage I’m at. the logging module in Python accepts text values for log levels, such as ‘DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’, and ‘CRITICAL’<\/p>\n
\r\nimport sys\r\nimport logging\r\nfrom awsglue.utils import getResolvedOptions\r\n\r\ndef setup_logger(level='INFO'):\r\n numeric_level = getattr(logging, level.upper(), None)\r\n if not isinstance(numeric_level, int):\r\n raise ValueError(f'Invalid log level: {level}')\r\n\r\n logging.basicConfig(\r\n level=numeric_level, \r\n format='%(asctime)s [%(levelname)s] - %(funcName)s: %(message)s',\r\n datefmt='%Y-%m-%d %H:%M:%S'\r\n )\r\n return logging.getLogger(__name__)\r\n\r\n# Read log level from the job arguments\r\nargs = getResolvedOptions(sys.argv, ['LOG_LEVEL'])\r\nlog_level = args['LOG_LEVEL']\r\n\r\nlogger = setup_logger(log_level)\r\n\r\ndef sample_function():\r\n logger.debug("This is a debug message.")\r\n logger.info("This is an info message.")\r\n logger.error("This is an error message.")\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"Working on various Python projects has taught me the importance of consistent logging, especially when dealing with distributed computing frameworks like Spark. Logging is not just about keeping track of errors or information; it’s about having a detailed and systematic record of the operations to understand the flow of your program and quickly diagnose issues.… Continue reading My Python logging setup<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":2400,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[331],"tags":[341],"class_list":["post-2399","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-software-development-en-en","tag-python","entry"],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2023\/08\/5aadc010-b913-4eb0-9388-cd8d6c743271.jpg","jetpack_shortlink":"https:\/\/wp.me\/p1tlzy-CH","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2302,"url":"https:\/\/www.danielpradilla.info\/blog\/aws-lambda-python\/","url_meta":{"origin":2399,"position":0},"title":"10 things I learned while deploying my first python function to AWS Lambda","author":"Daniel Pradilla","date":"04\/01\/2019","format":false,"excerpt":"I spent a few days on and off trying to deploy a Flask REST service to AWS Lambda, just to experience what the cool kids were talking about. These are some of the things I learned along the way: \u00a0 Zappa is the easiest packager\/deployer for python (as of December\u2026","rel":"","context":"In "Software Dev."","block_context":{"text":"Software Dev.","link":"https:\/\/www.danielpradilla.info\/blog\/category\/software-development-en-en\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2019\/01\/AWS-Lambda-290x300.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2224,"url":"https:\/\/www.danielpradilla.info\/blog\/pentaho-is-slow-for-servers-with-too-many-home-directories\/","url_meta":{"origin":2399,"position":1},"title":"Pentaho is slow for servers with too many home directories","author":"Daniel Pradilla","date":"28\/04\/2018","format":false,"excerpt":"Over the course of two years, browsing solutions on our Pentaho 5.4 server became progressively slow. It came to a point in which you had to wait 2-3 minutes to see the list of solutions in the Pentaho User Console. The catalina log didn't say much and we didn't have\u2026","rel":"","context":"In "Software Dev."","block_context":{"text":"Software Dev.","link":"https:\/\/www.danielpradilla.info\/blog\/category\/software-development-en-en\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2416,"url":"https:\/\/www.danielpradilla.info\/blog\/echotree\/","url_meta":{"origin":2399,"position":2},"title":"EchoTree","author":"Daniel Pradilla","date":"03\/02\/2026","format":false,"excerpt":"For years now I kept some of my social accounts alive with a Python script. It read from a collection of RSS feeds and shared at random times. It worked, my feeds stayed active, and people asked me what was I doing reading at 3am. The script had a bit\u2026","rel":"","context":"In "Best of"","block_context":{"text":"Best of","link":"https:\/\/www.danielpradilla.info\/blog\/category\/bestof\/"},"img":{"alt_text":"EchoTree","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2026\/02\/echotree.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2026\/02\/echotree.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2026\/02\/echotree.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2026\/02\/echotree.jpg?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":2205,"url":"https:\/\/www.danielpradilla.info\/blog\/linear-optimization-with-or-tools\/","url_meta":{"origin":2399,"position":3},"title":"Linear Optimization with or-tools","author":"Daniel Pradilla","date":"07\/06\/2017","format":false,"excerpt":"\u00a0 Getting started Over the last couple of months I've been getting my feet wet with linear programming and mathematical optimisation. I got a sense of how it all worked from this Discrete Optimisation course in Coursera and googling around I discovered that there are a ton of tools out\u2026","rel":"","context":"In "Best of"","block_context":{"text":"Best of","link":"https:\/\/www.danielpradilla.info\/blog\/category\/bestof\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/06\/grocery_bag_brown_bag.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/06\/grocery_bag_brown_bag.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/06\/grocery_bag_brown_bag.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/06\/grocery_bag_brown_bag.jpg?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":2229,"url":"https:\/\/www.danielpradilla.info\/blog\/linear-optimization-with-or-tools-containerizing-a-gunicorn-web-application\/","url_meta":{"origin":2399,"position":4},"title":"Linear optimization with or-tools: containerizing a gunicorn web application","author":"Daniel Pradilla","date":"15\/05\/2018","format":false,"excerpt":"Previously, we left our app working with our local python+gunicorn+nginx installation. In order to get there we had to do quite a bit of configuration and if we wanted to deploy this in a server or send it to a friend, we would have to go through a very error-prone\u2026","rel":"","context":"In "Software Dev."","block_context":{"text":"Software Dev.","link":"https:\/\/www.danielpradilla.info\/blog\/category\/software-development-en-en\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2018\/05\/docker.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2018\/05\/docker.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2018\/05\/docker.jpg?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":2212,"url":"https:\/\/www.danielpradilla.info\/blog\/linear-optimization-with-or-tools-building-a-web-front-end-with-falcon-and-gunicorn\/","url_meta":{"origin":2399,"position":5},"title":"Linear Optimization with or-tools \u00e2\u20ac\u201d building a web front-end with falcon and gunicorn","author":"Daniel Pradilla","date":"14\/11\/2017","format":false,"excerpt":"In a previous post, I put together a script for solving a linear optimisation problem using Google's OR-tools. This python script is callable from the command line and you kinda need to know what you are doing and how to organize the parameters. So, in order to address this difficulty,\u2026","rel":"","context":"In "Software Dev."","block_context":{"text":"Software Dev.","link":"https:\/\/www.danielpradilla.info\/blog\/category\/software-development-en-en\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/11\/groceryshopping.gif?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/11\/groceryshopping.gif?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/11\/groceryshopping.gif?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.danielpradilla.info\/blog\/wp-content\/uploads\/2017\/11\/groceryshopping.gif?resize=700%2C400&ssl=1 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/posts\/2399","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/comments?post=2399"}],"version-history":[{"count":0,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/posts\/2399\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/media\/2400"}],"wp:attachment":[{"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/media?parent=2399"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/categories?post=2399"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.danielpradilla.info\/blog\/wp-json\/wp\/v2\/tags?post=2399"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}