# Nizar's Blog — Full Text > Personal technical blog by Nizar Selander on software development, AI and LLM workflows, web performance, and security. This file concatenates the full Markdown of every published post, newest first. Source: https://nizar.se · Feed: https://nizar.se/feed.xml · Index: https://nizar.se/llms.txt --- # Look Smarter With Git: Rewriting Your History URL: https://nizar.se/look-smarter-with-git-rewriting-your-history/ Date: 2022-08-08 As developers, we are encouraged to commit code changes early and frequently. A benefit of which is being able to revert to previous states when needed. This luxury, however, can come at the expense of a rather messy Git history; one that communicates how the intended changes were reached rather than what they are. This blog post is co-authored by [Raniz](https://raniz.blog/2022-08-08_look-smarter-with-git/). ## Intro As developers, we are encouraged to commit code changes early and frequently. A benefit of which is being able to revert to previous states when needed. This luxury, however, can come at the expense of a rather messy Git history; one that communicates how the intended changes were reached rather than what they are. This is considerably noticeable in projects with active main branches, a consequence of the additional merge commit to a feature branch's history every time upstream changes are incorporated into it. Which begs the question, should we strive for a more concise and readable history in favor of a truly representative one? We believe that a Git history that makes it easy for others to follow and understand the history of a project is a preferable one, a history that focuses on communicating a series of coherent, and logically separated changes. Luckily, Git provides us with the tools to do both. In this post, we present some of our favorite tools that we use to reorganize intermediate changes into coherent commits and in turn make the improvements we want to a project's history. ## Git Reset Git Reset is often used while working with our staging area; this area is where we add the changes that we commit when using `git commit`. Git reset can be used to rewind history in two ways, by discarding our changes and by keeping them intact. ### Hard Reset A hard reset is used to rewind history and discard changes. That is, if we no longer wish to keep changes made in a previous commit, we can run `git reset --hard HEAD~1` and have Git rewind our branch to the preceding commit. ### Soft Reset A soft reset works the same way as a hard reset except that it keeps our changes. That is, changes made in commits that we rewind will be kept in the staging area. This allows us to selectively pick the changes that we want into one or more new commits. ## Git Add Patch The Git Add command can be used to not only add entire files to our staging area, but to also selectively pick lines from one or more changed files. The `git add -p [file]` command contains an optional argument for the path to a file which contains changes in our repository. If no file path is provided, Git will work with every file in the repository that contains changes. That is, Git will iterate through all the changes and will, for each change, ask whether to add it to the staging area or not. ## Git Rebase Git Rebase is used to move one or more commits from one place in the Git history to another. For example: Let's say that we have a project with a main branch and a feature branch, both of which have two commits added to them after the branching point as illustrated in the figure below. ![Git branching](/uploads/git-branch.svg) ### Regular Rebase A regular rebase is often used instead of a merge. To illustrate how it works and how the two differ, we look at how a merge works. When two branches are merged, a new commit is created. This new commit is the intersection point of two separate histories. If there are any conflicts between the two histories, they are resolved in the merge commit and added to the history. ![Git merge](/uploads/git-merge.svg) Rebasing achieves the same goal but does so in a different way. When a branch is rebased, its start in history is moved from one spot in history to another, most often in front of the last commit on the branch we would have merged with. That is, we place one branch after the other in history instead of intersecting two branches. Conflicts are resolved during rebasing and commits containing conflicting changes are modified to no longer be in conflict, effectively removing the conflicts from history. ![Git rebase](/uploads/git-rebase.svg) As such, a rebase creates a linear history, one that is easier to follow. This is also visible when viewing the history using a graphical tool; the history looks like a straight line rather than a set of intertwining parallel tracks. ### Interactive Rebase An interactive rebase allows us to take full control of the history. This is accomplished by invoking the `git rebase -i ` command. `startpoint` here is a reference to a commit, such as a branch or a commit-SHA, in the history from where we want to start making changes. Running the command creates a text file that is opened by an editor, allowing us to make changes to the commit history. Editing this file allows us to, among other things: - Reorder commits. - Remove a commit. - Combine (squash) commits. - Edit a commits' message. - Stop a rebase process, start a shell, and edit the contents of a commit before proceeding. Together, and with the tools and techniques described above, we are able to to reorder, combine, split, and edit commits to clean up the history. We can do this once our code is at a point where we think it is ready for for others to see. ## When Not To Mess With History It is important to know that you should never rebase a public branch. That is, you should not rewrite the branch history if someone else shares it. The main branch is an example of such a branch. Doing so will result in a situation that is tricky to resolve and leaves a confusing history. That said, you are free to do so on your development branches before creating pull requests. In fact, and as discussed in this post, we believe that such efforts provide valuable improvements for everyone using the project. --- # Serving Sites with NGINX QUIC URL: https://nizar.se/serving-sites-with-nginx-quic/ Date: 2023-02-28 A quick look at QUIC and how to use NGINX’s newly released pre-built packages to enable it for your site. NGINX has [recently released](https://www.nginx.com/blog/binary-packages-for-preview-nginx-quic-http3-implementation/) prebuilt [binary packages](https://quic.nginx.org/packages.html) for the preview NGINX QUIC+HTTP/3 implementation for Red Hat Enterprise Linux 9 and Ubuntu 22.04. The prebuilt binary packages eliminate the need to compile from source and automatically install a [quicktls](https://github.com/quictls) library package as a dependency as OpenSSL does not yet support QUIC. ## QUIC QUIC is a general-purpose transport layer network protocol that provides built-in security and improved performance compared to TCP + TLS. Its built-in security features, such as encryption and authentication, allow for the exchange of setup keys and protocols to take place in the initial handshake. Thus reducing the connection setup overhead and latency as shown by the diagram below. ![QUIC diagram](/uploads/quic-handshake-comparison.gif) For a more detailed breakdown of QUIC and how it works, checkout Cloudflare's blog article [The Road to QUIC](https://blog.cloudflare.com/the-road-to-quic/). ## Using NGINX QUIC What follows is a step-by-step guide on how to serve a website using NGINX QUIC. For this setup, we will use a newly created Ubuntu 22.04 server. ### Update the system We start by ensuring that the system is up-to-date as usual. ```shell # Install the latest updates sudo apt update && sudo apt upgrade -y # Reboot if necessary sudo reboot -h now ``` ### Install NGINX-QUIC We then set up the repository and install the pre-built packages. ```shell # Install the prerequisites sudo apt update && sudo apt install curl gnupg2 ca-certificates lsb-release ubuntu-keyring # Import an official nginx signing key to verify packages authenticity curl https://nginx.org/keys/nginx_signing.key | gpg --dearmor \ | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null # Set up the apt repository for nginx-quic packages echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ https://packages.nginx.org/nginx-quic/ubuntu `lsb_release -cs` nginx-quic" \ | sudo tee /etc/apt/sources.list.d/nginx-quic.list # Install nginx-quic sudo apt update sudo apt install nginx-quic ``` ### Firewall Enable the firewall if it is not already enabled and make sure to allow UDP traffic through port 443. UDP uses a connectionless communication model, it is a fire and forget protocol. It is what makes the reduction of overhead possible. ```shell # Adjust firewall sudo ufw default allow outgoing sudo ufw default deny incoming sudo ufw allow ssh sudo ufw allow 80/tcp sudo ufw allow 443/tcp sudo ufw allow 443/udp # Enable firewall sudo ufw enable ``` ### Certbot Install certbot, or your favorite tool to issue and renew certificates. A lit of alternative clients can be found [here](https://letsencrypt.org/docs/client-options/). We will use snap to since it is the method recommended by Certbot. You can find alternative installation methods [here](https://eff-certbot.readthedocs.io/en/stable/install.html). ```shell # Make sure snapd core is up to date sudo snap install core; sudo snap refresh core # Ensure that no older version is already installed sudo apt remove certbot # Install the certbot package sudo snap install --classic certbot # Link the certbot command from the snap install directory sudo ln -s /snap/bin/certbot /usr/bin/certbot ``` ### Site directory We will now create a directory to store our site and its data. This is where the server will look for the site's contents when serving out visitors. You can choose a different location that the one I chose, just make sure to make the same adjustment in the other places that this path appears in as you follow along. ```shell sudo mkdir -p /var/www/WEBSITE/html/ sudo chown -R www-data:www-data /var/www/WEBSITE/html/ sudo chmod -R 755 /var/www/WEBSITE ``` ### Webpage With our directory created, we will now create a simple html page for the purpose of demonstrating the functionality. This can be replaced with your actual site contents when we are done. ```shell # Create a sample index.html nano /var/www/WEBSITE/html/index.html ``` Paste in the following snippet and save. ```html Welcome to WEBSITE!

Success! The WEBSITE server block is working!

``` ### Configure NGINX It is now time to make some adjustments to our NGINX configuration. ```shell sudo nano /etc/nginx/nginx.conf ``` Adjust the config file to match the following block, which is enough to get us started for now. ```txt user www-data; worker_processes auto; error_log /var/log/nginx/error.log notice; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; gzip on; include /etc/nginx/conf.d/*.conf; } ``` What follows is a brief explanation of what the different directives and values do. - `user` defines the user used by the worker processes. - `worker-processes` the number of worker processes. Setting it to the number of CPU cores is a good start, `auto` automatically detects it for us. - `error_log` defines a log file to store logs and the level of logging. - `pid` defines a file that will store the process ID of the main process. - `events` configuration of connection processing. - `worker_connections` sets the maximum number of simultaneous connections that can be opened by a worker process. - the top-level `http` block. - `include /etc/nginx/mime.types` tells browsers how to handle different file formats. - `default_type application/octet-stream;` tells browsers to treat files not identified in `/etc/nginx/mime.types` as downloadable binaries. - `log_format` specifies the log format. - `access_log` sets the path and format for logging. - `send_file` ensures that nginx operations will not block disk I/O. - `keepalive_timeout` the duration to keep worker_connections open for each client. - `gzip` compress data to browsers to enhance performance. - `include /etc/nginx/conf.d/*.conf;` include all configuration files in provided directory. ### Create the website configuration We will now create a configuration file for the website in the /etc/nginx/conf.d/ directory inline with the new conventions which you can read more about [here](https://www.oreilly.com/library/view/nginx-cookbook/9781492049098/ch01.html). All .conf files placed in this directory are included in the top-level http block. ```shell sudo nano /etc/nginx/conf.d/WEBSITE.conf ``` Paste in the following content and save the file. ```txt server { listen 80; server_name WEBSITE www.WEBSITE; root /var/www/WEBSITE/html; } ``` What follows is a brief explanation of what the different directives and values do. - `server` defines a new server block for nginx to listen to. - `listen 80` the port nginx will listen on. - `server_name` the hostnames of the requests which should be directed to this server. - `root` tells nginx where to look for content. ### Start NGINX It's time to apply the changes and start NGINX. ```shell # Start and enable nginx on system startup if not already enabled sudo systemctl start nginx sudo systemctl enable nginx # Check the status of nginx sudo systemctl status nginx # Make sure no errors were encountered sudo nginx -t ``` Makes sure that no errors were encountered. The last command should display helpful information to help you troubleshoot any failed validations. ### Generate certificates With NGINX up and running, it is time to generate our certificates. We do this with the help of certbot. When prompted, fill in your email address and agree to the terms. ```shell sudo certbot --nginx -d WEBSITE -d www.WEBSITE ``` Those certificates are valid for 90 days. Certbot adds a systemd timer that checks our certs for us twice a day and renews any certs that will expire within 30 days. We can verify the status of the timer by running the following command. ```shell sudo systemctl status snap.certbot.renew.service ``` We can also test the renewal process by running the following command. ```shell sudo certbot renew --dry-run ``` Let's Encrypt will also send you a warning email to the email account you provided when creating the certificates if the renewal process fails. ### Enable QUIC Certbot should now have generated the certificates for us and updated our site's configuration file accordingly. We need to make some final adjustments to enable QUIC. ```shell sudo nano /etc/nginx/conf.d/WEBSITE.conf ``` Adjust the file's contents to match the following block. ```txt server { # Using the same port number for QUIC and TCP listen 443 http3 reuseport; # IPv4 QUIC listen 443 ssl http2; # IPv4 TCP listen [::]:443 http3 reuseport; # IPv6 QUIC listen [::]:443 ssl http2; # IPv6 TCP # Server name server_name WEBSITE www.WEBSITE; # Site root root /var/www/WEBSITE/html; # Certificates ssl_certificate /etc/letsencrypt/live/WEBSITE/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/WEBSITE/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot } server { if ($host = www.WEBSITE) { return 301 https://$host$request_uri; } # managed by Certbot if ($host = WEBSITE) { return 301 https://$host$request_uri; } # managed by Certbot listen 80; server_name WEBSITE www.WEBSITE; return 404; # managed by Certbot } ``` Here, we added `listen [::]` so that nginx listens to IPv6 connections. You can remove this directive if you have not enabled IPv6 for your domain. We have also configured `HTTP/2` to be the starting http version for new connections instead of `HTTP/1.1` for better performance. Connections will switch to QUIC after it is discovered. The second server block is used to redirect unencrypted traffic to encrypted traffic. _Update_: The configuration has changed. Use `quic` and `ssl` separately in `listen` directives and enable `http2` globally with `http2 on;`. Additionally, new [QUIC-related settings](https://nginx.org/en/docs/quic.html) have been added: ```txt # Using the same port number for QUIC and TCP listen 443 quic reuseport; listen 443 ssl; listen [::]:443 quic; listen [::]:443 ssl; http2 on; # QUIC quic_retry on; ssl_early_data on; quic_gso on; ``` ### Apply changes It is time to apply the new changes after checking that everything is fine. ```shell # Make sure no errors were encountered sudo nginx -t # Restart NGINX sudo systemctl reload nginx ``` ### Live! The website should now be live with QUIC+HTTP/3 enabled. ![View of website in browser](/uploads/configuring-quic-browser-result.gif) ### Verify Head over to [https://www.http3check.net/](https://www.http3check.net/) to verify that QUIC and HTTP/3 are supported on your site. ![QUIC verification](/uploads/quic-support-verification.gif) ## Concluding Remarks With this _quic_ demonstration completed, there are some things to consider before using it. Given that internet service has gotten more reliable over the years, the likelihood of issues caused by dropped packages has become increasingly unlikely. At the same time, the amount of bandwidth saved makes it an attractive tradeoff especially on the server side. This is also a preview release. That said, there are several production deployments according to [NGINX](https://quic.nginx.org/). I will also cover topics such as optimising NGINX for performance and strengthening security through the use of headers in future posts. --- # Improving Website Speed with Compression URL: https://nizar.se/improving-website-speed-with-compression/ Date: 2023-06-26 Compression plays a vital role in the modern web. It helps us deliver content faster to users while preserving network resources. With technologies like Gzip and Brotli, we can make websites more efficient without compromising the quality of the data. In this post, we will explore how to improve website speed by leveraging HTTP compression and pre-compressing files. Google engineers, Arvind Jain and Jason Glasgow, stated that more than 99 human years are wasted every day because of uncompressed content [[1]]. That was in 2009. Advancements in technology have since then brought improvements to internet speed, connectivity, and coverage. However, a lot has also changed on the web and how we use it. Between 2012 and 2022, the average desktop page size increased from 803 KB to 2,284 KB. For mobile, it increased from 386 KB to 2,010 KB. A staggering increase of 184% and 420% respectively in a single decade [[2]]. During the same period, the number of internet users has also more than doubled, increasing from roughly 2.2 to 5.3 billion internet users, an increase from roughly 35% to 66% of the world population that use the internet [[3]]. At the risk of preaching to the choir, the minute performance improvements we make to the websites we create accumulate to astounding and far-reaching impacts. Their importance and significance can not be understated, especially considering how simple some of them are. In this post, we'll explore how to improve website speed by leveraging HTTP compression and pre-compressing files. ## HTTP Compression One technique that has significantly contributed to website speed improvement is HTTP compression. This seamless capability in our web servers and browsers, which can easily go unnoticed, improves transfer speed and bandwidth utilization by compressing HTTP data before sending it. The scheme used for the compression is negotiated through the client advertising methods it supports. ```text GET /encrypted-area HTTP/1.1 Host: www.example.com Accept-Encoding: gzip, deflate ``` The server can also support multiple compression schemes. In such a case, the server lists them in the `Content-Encoding` or `Transfer-Encoding` field in the HTTP response. ```text HTTP/1.1 200 OK Date: mon, 26 June 2016 22:38:34 GMT Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux) Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT Accept-Ranges: bytes Content-Length: 438 Connection: close Content-Type: text/html; charset=UTF-8 Content-Encoding: gzip ``` ## Compression Schemes The two most common compression schemes used today are Gzip (`gzip`) and Brotli (`br`). Gzip, the older of the two, was released in 1993 during the formative years of the internet. It was created to compress files and has since been adapted to compress streams. Brotli on the other hand, which was released in 2013 by Google, was designed from the ground up to compress streams making it a preferred scheme for web. Both Gzip and Brotli combine variations of the [LZ77](https://en.wikipedia.org/wiki/LZ77_and_LZ78) algorithm and [Huffman coding](https://en.wikipedia.org/wiki/Huffman_coding), among others, to provide lossless general-purpose compression. LZ77 replaces repeated occurrences of data with references to a single copy of it using an encoded length-distance pair. This match effectively states that the length following characters are identical to the characters distance before them. This algorithm formed the basis of compression schemes such as GIF and the deflate algorithm used in PNG and ZIP. The Huffman coding algorithm further compresses the data by using a dictionary of recurring characters and their frequency. Characters, and groups of which, are replaced by variable-length codes based on their occurrence with more frequent ones being assigned shorter codes. Brotli is further able to reach higher compression density by utilizing context models and other improvements. It also uses a pre-defined dictionary along its dynamic memory, roughly 120 KiB containing 13,000 common words, phrases and substrings derived from internet documents. This allows Brotli to reach compression densities that are roughly 15% denser for JavaScript and CSS files, and 20% denser for HTML files compared to Gzip. That said, Brotli can also take longer to compress data compared to Gzip. The higher the level of compression, the longer it takes. ## Pre-Compressing Files While compression significantly reduces the size of transmitted data, it also happens at run time and as such impedes performance due to the required processing overhead. Luckily, we can utilize modules that allow us to serve pre-compressed files, effectively eliminating all the processing overhead at runtime. This is ideal for serving static content and static files. We will have to first compress our files. The `node:zlib` module provides all the compression functionality needed to do so using Gzip and Brotli. The documentation provides an example of how to accomplish this by piping the source stream (file) through a `zlib` Transform into a destination stream. ```javascript const { createGzip } = require('node:zlib') const { pipeline } = require('node:stream') const { createReadStream, createWriteStream } = require('node:fs') const gzip = createGzip() const source = createReadStream('input.txt') const destination = createWriteStream('input.txt.gz') pipeline(source, gzip, destination, (err) => { if (err) { console.error('An error occurred:', err) process.exitCode = 1 } }) ``` For Brotli, we simply replace the Gzip object with a `BrotliCompress` object. We can call `createBrotliCompress()` instead of `createGzip()` to create our transform stream. Additionally, we can pass an `options` object to specify the compression level and fine-tune other parameters to our liking. We then simply create the necessary functionality to recursively iterate through the files in our build directory and create `.gz` and `.br` compressed files next to the uncompressed files. Finally, we add the command to our build script and have it automatically execute as a final build step. ## Serving Pre-Compressed Files For NGINX, we can use [ngx_http_gzip_static_module](https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html#gzip_static) for Gzip and [ngx_brotli](https://github.com/google/ngx_brotli#brotli_static) for Brotli precompressed files. These allow us to precompress our files and have them served as such. Nginx uses a nifty tool called [pkg-oss](https://hg.nginx.org/pkg-oss/) to create the dynamic module packages for their official repositories. It contains a script that automates the process, which we can use to create installable packages with the correct dependencies for our modules. This method ensures that the dependency between our modules and NGINX is honored, facilitating seamless upgrades and avoiding upgrade failures. The code block below shows how we can use it to create `deb` packages for our `ngx_brotli` module. You can use `curl` or any other similar tool instead of `wget`. ```text wget https://hg.nginx.org/pkg-oss/raw-file/default/build_module.sh chmod a+x build_module.sh ./build_module.sh -v 1.23.3 --force-dynamic https://github.com/google/ngx_brotli.git ``` Do note however that this should be done in a separate build environment to ensure that only operation-necessary software is installed on the production server. Further, this script might not work for every module and the installable packages created by it are not intended for distribution. Finally, seamless upgrades require a `yum` or `apt` repository. For instructions and more information, check out Liam Crillys article [Creating Installable Packages for Dynamic Modules](https://www.nginx.com/blog/creating-installable-packages-dynamic-modules/). Upon completion, the script outputs the path of the created `deb` packages which we can install with the help of `dpkg`. ```shell dpkg -i nginx-module-brotli_1.23.3+1.0-1~jammy_amd64.deb ``` We then need to configure NGINX to use the newly installed module. We do this by editing the main configuration file. ```shell nano /etc/nginx/nginx.conf ``` We simply add the directive we are interested in to the http block. For example: ```text brotli on; brotli_comp_level 6; brotli_static on; brotli_types application/atom+xml application/javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/x-javascript application/xhtml+xml application/xml font/eot font/opentype font/otf font/truetype image/svg+xml image/vnd.microsoft.icon image/x-icon image/x-win-bitmap text/css text/javascript text/plain text/xml; ``` - `brotli`: Enables on-the-fly compression of responses. - `brotli_comp_level`: Sets the on-the-fly compression levels. Range is 0 - 11, default is 6. - `brotli_static`: Enables checking for pre-compressed files with .br extension. - `brotli_types`: MIME types to enable on-the-fly compression for. Check the official [ngx_brotli](https://github.com/google/ngx_brotli) repo for additional information and configuration. Check for any syntax errors after saving with the following command: ```shell nginx -t ``` The output should look something like this: ```text nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful ``` Finally, restart NGINX to apply the changes. ```shell systemctl restart nginx ``` NGINX is now configured with Brotli support. You can verify this by running the following command: ```text curl -H ‘Accept-Encoding: br’ -I https://your-website.com ``` The result should contain the following line: ```text content-encoding: br ``` Be also sure to test your improved website speed by using PageSpeed Insights, Lighthouse or any other tool you prefer. Your numbers may vary depending on several factors such as existing bottlenecks, but you should see an improvement. ## Results After compressing and serving static files, the loading speed of my website improved significantly. However, don't just take my word for it. Let's examine some real-world examples that illustrate the effectiveness of compression. Here is how the time-to-interactive changed on mobile for the following sites: - Organization website: Reduced from 1.5 to 0.9 seconds. - Conference website: Reduced from 1.4 to 1.0 seconds. These are just a few examples of how powerful compression can be in improving website speed. With the easy implementation and significant benefits, it is clear why HTTP compression and pre-compressing files have become standard practice in web development. ## Conclusion Compression plays a vital role in the modern web. It helps us deliver content faster to users while preserving network resources. With technologies like Gzip and Brotli, we can make websites more efficient without compromising the quality of the data. As the internet continues to grow and evolve, we can only expect compression techniques to become even more critical. In future posts, we'll explore more ways to improve website performance and make the most of available resources. ## References [1]: https://developers.googleblog.com/2009/11/use-compression-to-make-web-faster.html 'Use compression to make the web faster' [2]: https://www.keycdn.com/support/the-growth-of-web-page-size 'The Growth of Web Page Size' [3]: https://www.itu.int/hub/publication/d-ind-ict_mdd-2022 'Measuring digital development: Facts and Figures 2022' 1. [https://developers.googleblog.com/2009/11/use-compression-to-make-web-faster.html](https://developers.googleblog.com/2009/11/use-compression-to-make-web-faster.html) 2. [https://www.keycdn.com/support/the-growth-of-web-page-size](https://www.keycdn.com/support/the-growth-of-web-page-size) 3. [https://www.itu.int/hub/publication/d-ind-ict_mdd-2022](https://www.itu.int/hub/publication/d-ind-ict_mdd-2022) --- # Striving for Performance and Design Harmony in Web Development URL: https://nizar.se/striving-for-performance-and-design-harmony-in-web-development/ Date: 2023-09-20 In modern web development, a fine balance between performance and user-centric design is key. It doesn't just enhance visitor retention; it deepens their interaction with the content. My recent project provided me a deep dive into achieving this delicate balance. Here is a recount of my experience and the insights that I’ve gained. In modern web development, a fine balance between performance and user-centric design is key. It doesn't just enhance visitor retention; it deepens their interaction with the content. My recent project provided me a deep dive into achieving this delicate balance. Here is a recount of my experience and the insights that I've gained. ## The Task at Hand The project involved transforming a site initially built using 11ty and Forestry CMS. The goal was clear: transitioning to Cloudcannon and securing top-notch performance metrics across the board. ![Website score on webperf](/uploads/webperf-report-for-factor10.gif) _The website's final scores for accessibility, speed, standards, privacy, and security. Securing it the top position of all websites in Sweden measured by [webperf.se](https://webperf.se/toplist/)._ ## Crafting the Developer Experience A well crafted website not only looks good but is also built upon pillars of performance, accessibility, and best practices. Here, tools like [Tailwind](https://tailwindcss.com/) offer an appealing promise of swift development. However, every tool has its caveats. In the case of Tailwind, it produced CSS with [custom properties](https://developer.mozilla.org/en-US/docs/Web/CSS/Using_CSS_custom_properties), a feature that at that point was yet to be [standardized](https://drafts.csswg.org/css-variables/). Instead of letting this be a setback, I explored alternatives to establish an equally efficient and issue-free developer environment. ## Embracing Modular CSS Let's dive a bit into the CSS. A modular CSS structure promises maintainability and scalability in design, paving the way for rapid feature deployment and updates. It ensures that while each component stands on its own, it is harmonious with overarching design. The structure we employed embodies this principle: ``` component-library/ ├─ components/ │ └─ sample/ │ ├─ sample.bookshop.yml │ ├─ sample.test.js │ ├─ sample.scss │ └─ sample.eleventy.liquid └─ shared/ └─ styles/ ├─ global.scss └─ variables.scss ``` But what makes this structure dynamic? Enter SCSS. A powerful extension of the traditional CSS, SCSS bestows developers with features like variables and nested rules. While vanilla CSS has its merits, SCSS allows for a more dynamic and fluid styling approach. Our build process uses tools like [Sass](https://www.npmjs.com/package/sass) to consolidate and optimize SCSS files, ensuring seamless style deployment. The journey from SCSS to optimized CSS follows these steps: - Identify the SCSS files. - Compile them into regular CSS. - Optimize the resultant CSS. - Store it for deployment. ### Discovery To begin the journey of styling optimization, we must first pinpoint where all the styles reside. Utilizing [fast-glob](https://www.npmjs.com/package/fast-glob), we efficiently traverse the file system, seeking out the SCSS files. Imagine wanting to identify all `.scss` and `.css` files within the `src` directory. Here is how it would look: ```js const fg = require('fast-glob') const entries = await fg('src/**/*.{css,scss}') ``` ### Compilation Having identified our list of SCSS files, the next step is compilation. Conventionally, you'd find a "main" SCSS file that `@import`s other SCSS files, setting up a clear hierarchy. Compiling this main file processes all the imported files in the order of their appearance. ```js const sass = require('sass') const output = sass.renderSync({ file: 'path/to/main.scss' }) ``` A more flexible alternative to the traditional method is dynamic compilation, as exemplified by the Bookshop sass bundler: ```js const sassInput = entries.map((file) => `@import "${file}";`).join('\n') const output = sass.renderSync({ data: sassInput }) ``` Here, we create an in-memory virtual SCSS file that imports every SCSS file detected. By compiling it, we incorporate all unique component styles. ### Post-Processing Compilation, while crucial, is just one part of the equation. We must enhance and optimize the compiled styles. This is where powerful tools like [PostCSS](https://www.npmjs.com/package/postcss) and [cssnano](https://www.npmjs.com/package/cssnano) come into play. PostCSS is a tool that doesn't merely refine styles — it transforms them through its plugin-based system, allowing developers to decide exactly which transformations they want to apply. The [Autoprefixer](https://www.npmjs.com/package/autoprefixer) plugin is a prime example, appending vendor prefixes based on [Can I use](https://caniuse.com/) data, ensuring consistent cross-browser compatibility. While PostCSS transforms, cssnano tightens. It compresses, strips, and trims the CSS into its leanest form. ```js const postcss = require('postcss') const cssnano = require('cssnano') const autoprefixer = require('autoprefixer') const inputCSS = 'a { transition: transform 1s; }' // Simple CSS for demonstration. const processedCSS = await postcss([autoprefixer, cssnano]).process(inputCSS) const finalCss = processedCSS.css ``` Together, PostCSS and cssnano aim to make our CSS not only adaptable but also blazing fast. ### Writing Having navigated the maze of style optimization, our last step is simple: writing our refined and optimized CSS to a file. ## Render Time Ensuring fast page rendering is crucial for a seamless user experience. To realize this, we must permit the browser to begin rendering even as resources are still in the loading phase. The primary challenge lies in eliminating layout shifts and avoiding any sudden visual disturbances. For optimal performance, resource loading and deployment must be carefully managed, particularly given the potential influence on the page during their loading phase. ### Font display While custom fonts can undeniably elevate a website's aesthetic appeal, they may impede rendering speed. Yet, with the technique of font swapping, it's possible to strike a balance between aesthetics and efficiency. By integrating font swapping in our CSS, the browser is directed to utilize a system or fallback font while the page is still loading. This circumvents the scenario where users find themselves gazing at an empty space. Once the custom font has been fully loaded, the browser subtly switches to it, ensuring that the design's integrity remains uncompromised without any delay for the user. The [font-display](https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face/font-display) descriptor facilitates this: ```css font-display: swap; ``` Enhancing font loading further, preloading the font primes the browser early about its significance, allowing for efficient downloading. ```html ``` The above method achieves this without introducing any JavaScript to our site. ## Critical CSS The term 'above-the-fold' originates from newspaper design, referring to the content immediately visible when a page loads, without the need for scrolling. By prioritizing the delivery of the critical CSS necessary for this portion, we ensure the user's first impression is swift and seamless. We do this by inlining these critical styles directly into the HTML's ``, and thereby allowing the browser to immediately render the immediate content without having to wait for the entire stylesheet to load. The npm package, [critical](https://www.npmjs.com/package/critical), offers a simplified solution: ```js const critical = require('critical') critical.generate({ inline: true, // Will inline the generated CSS into the HTML base: 'dist/', // Your base directory src: 'index.html', // Source HTML file to be processed width: 1300, // Viewport width height: 900, // Viewport height ignore: { // Rules to ignore, e.g., font-face which can be hefty atrule: ['@font-face'], }, }) ``` In the above example, the `index.html` file is analyzed to determine the styles needed for its above-the-fold content, and are subsequently inlined. This strategy provides several benefits. It accelerates the initial content rendering, allowing earlier user interaction, and side-steps render blocking delays. ## Layout shifts Responsive design has revolutionized how sites adjust across devices, but it's not without hurdles. For instance, while strategies like asynchronous content loading shows a lot of 'promise' — pun very much intended — it can occasionally result in unexpected layout shifts. Such shifts disorient users, leading to unintended clicks and detracting from a seamless browsing experience. A hands-on approach is to establish size constraints for elements that might adjust with incoming content. As an example, if a font swap alters an element's dimensions, the application of `min-height` and `min-width` becomes a vital safeguard. This strategy ensures that our content is primed to adapt to any changes, be they CMS updates or device reconfigurations, whilst maintaining a steadfast structure. ## Responsive Images As screen sizes become more diverse, the need for responsive image techniques grows. Choosing the right image size not only boosts a website's visual appeal but also ensures optimal performance. Here, the HTML attribute, `sizes`, stands out. The `sizes` attribute of the `picture` element allows us to guide the browser on the image display size for varying viewport conditions. By defining both upper and lower limits with this attribute, we ensure the best-suited image variant gets loaded. For instance, with the `sizes` attribute, we can dictate how the image adapts across different device widths. Here is how it works: ```html (min-width: 1280px) 426px, (min-width: 768px) and (max-width: 1280px) 33vw, (min-width: 640px) and (max-width: 768px) 50vw, (max-width: 640px) 100vw ``` - For screens wider than 1280 pixels, the image will be displayed at a fixed width of 426 pixels. - For screens between 768 and 1280 pixels, the image will take up 33% of the viewport width. - On screens ranging from 640 to 768 pixels, the image will cover half the viewport width. - For a screen width of 640 pixels or fewer, the image will span the entire viewport width. Alongside the `sizes` attribute, it's worth highlighting the value of `srcset`. While we dictate display conditions with `sizes`, `srcset` lists multiple image versions, letting the browser select the optimal one based on device specifics. The result is crisp visuals across devices, faster page loads, saved energy, and reduced bandwidth use. ## Security At first glance, embedding inline CSS directly into HTML appears to be a straightforward approach to styling. Yet, this simplicity can be deceptive, as it opens up potential security vulnerabilities and challenges, particularly when the CSS undergoes changes during development. To counter such vulnerabilities, modern browsers have incorporated a security feature called the Content Security Policy (CSP) Header. This feature delineates where various content types, be it scripts, styles, or images, can be sourced from, ensuring that only content from trusted sources gets executed or rendered. The significance of CSP becomes even clearer when we consider threats like cross-site scripting. This nefarious tactic allows attackers to inject code into web pages that unsuspecting users then view. Not only can it jeopardize user data, but it can also transform reputable sites into hubs of malicious activity. The CSP stands as a safeguard against these and other similar threats. Now, when one chooses to go the route of inlining CSS, a specific hash of the content needs to be added to the CSP header, signaling to the browser that this content is legitimate. But there is a catch: should the styling undergo any alterations during development, which is often the case, the associated hash needs a recalculation and subsequent update on the server. This step, often done manually, is both labor-intensive and susceptible to errors. A possible workaround is the use of a nonce — a unique, one-time token. However, this method demands a dynamic setting, such as a Node server. Given that our platform is tailored to serve only static files, venturing into such complexities is unwarranted. ## Automation We can make this process smoother and hands-free by utilizing a Python script during deployment: 1. First, we scan for HTML files: Store the deployment in a processing directory and sift through it, identifying the HTML files. 2. Detect Inline Styles: The script then parses the HTML files for inline styles. 3. Hash Calculation: For each detected style, the script computes its hash. 4. Config File Generation: Using a predefined template, the script produces a new configuration file for the site. Within this file, a placeholder is replaced with the calculated `style-src` hashes. 5. Deployment and Validation: A critical step in the process. Before pushing any changes, the script runs a series of tests and checks, validates the syntax and other parameters. Only when the validation succeeds are the deployment files sent to their designated destination. Subsequently, the configuration file is refreshed, and the server reloads it. 6. Cleanup: If tests uncover issues or anomalies, all interim changes within the processing directory are discarded without affecting the live website. This allows us to safeguard our site's security while also eliminating manual intervention that can be tedious and error-prone. ![Headers security summary showing an A+ rating](/uploads/factor10-headers-security-report-summary.gif) _Summary of the headers security analysis from [securityheaders.com](https://securityheaders.com/)_ ## Putting it all together By harnessing these strategies, we create sites that excel both visually and functionally. In the end, the elegance of a website isn't just in its visible design but also in its underlying mechanics. As designers and developers, our goal is to master both fronts, crafting experiences that are not only beautiful but are also performant, secure, and sustainable. With this, I would like to thank my colleagues at factor10 from whom I learn every day. I would also like to thank my co-workers [Anders](https://www.linkedin.com/in/wengelin/) and [Peter](https://www.linkedin.com/in/okkido/) who have worked alongside me for their excellent work. Feel free to check out the result for yourself: [factor10.com](https://factor10.com/) --- # Robot Control with the Jetson Nano URL: https://nizar.se/robot-control-with-the-jetson-nano/ Date: 2024-04-02 Two years ago, I got to dive into a project at the intersection of machine learning and hardware - right before ChatGPT became a household name and when terms like overfitting puzzled me. Nonetheless, I was really excited about it, especially because it involved both software and hardware, a combination Im quite fond of. Two years ago, I got to dive into a project at the intersection of machine learning and hardware - right before ChatGPT became a household name and when terms like [overfitting](https://en.wikipedia.org/wiki/Overfitting) puzzled me. Nonetheless, I was really excited about it, especially because it involved both software and hardware, a combination I'm quite fond of. The project was part of the [Nvidia Jetson AI Certification program](https://developer.nvidia.com/embedded/learn/jetson-ai-certification-programs), which challenges participants to demonstrate their machine learning skills using the [Jetson Nano Developer kit](https://developer.nvidia.com/embedded/jetson-nano-developer-kit). Essentially, the kit is an embedded computer designed specifically to accelerate machine learning applications. Though not required, I also had access to a [robot](https://www.waveshare.com/jetank-ai-kit.htm), which I was eager to use. I set a goal to train a model that would enable the robot's arm to be controlled via hand gestures. The robot was equipped with a small camera on its front, perfectly positioned for this task. ## Image Classification I began with image classification, where the model is trained to identify and categorize images into specific classes. My goal was to recognize four hand gestures: - Thumbs up - Thumbs down - Fingers spread - Fingers together ![Image classification demo of hand grip and gestures](/uploads/classification-feed-demo.gif) This allowed me to command the robot's arm and grip movement using these gestures. However, it was somewhat hit-or-miss. The approach was basic, capable of only recognizing one gesture at a time, which limited the ability to control the arm's elevation and grip simultaneously. Moreover, the control was binary; It lacked the nuance to specify degrees of movement or grip tightness, posing a challenge for precise control. ![Demo of the robot control using image classification](/uploads/classification-grip-demo.gif) ## Image Regression Moving forward, I explored image regression. Unlike classification, this method involved training the model to pinpoint my hand's features in images, providing coordinates for my fingertips. The challenge then became translating these coordinates into accurate robot movements. ![Image regression demo of hand grip](/uploads/regression-feed-index.gif) A significant hurdle was accounting for the camera's depth perception, essential for distinguishing between fingertips being spread apart versus the hand moving closer to the camera. I experimented with measuring distances between various hand landmarks as proxies for depth perception, such as the distance from the thumb's top and the wrist's bottom corner, or between the tips of the index and pinky fingers. This approach, however, introduced additional complexity and new challenges, such as adjusting for the hand tilting and panning. Ultimately, the model's error margins and the camera's limitations led me to simplify my approach. I chose to track only the thumb and index finger while maintaining a constant distance from the camera. This simpler model achieved more predictable results and easier control. ![Demo of the robot control using image regression](/uploads/regression-day-7.gif) While the project may seem straightforward, it was incredibly insightful, making the seemingly complex field of machine learning feel much more attainable. Every so often, I reflect on this project and wonder if a simpler solution exists for accurately considering distance. An intriguing possibility that comes to mind is the use of two cameras, mirroring the binocular vision of humans, to facilitate a more straightforward approach to depth perception. This idea excites me, and I'm keen to explore it further, possibly revisiting the project with a new perspective in the future. You can find the source code and assembly instruction on my GitHub [repo](https://github.com/nizos/jetarm). If you are curious about machine learning or robotics, I encourage you to give it a look - who knows what you might discover? --- # Optimizing GitHub Workflows for Efficiency and Sustainability URL: https://nizar.se/optimizing-github-workflows-for-efficiency-and-sustainability/ Date: 2024-06-12 Integrating automation into development workflows has become crucial in recent years. In this post, I'll share a few strategies I've found useful for making these workflows both more efficient and sustainable. We'll look into strategies such as canceling redundant jobs, setting appropriate workflow timeouts, optimizing caching techniques, and managing workflow triggers more effectively. Integrating automation into development workflows has become crucial in recent years. In this post, I'll share a few strategies I've found useful for making these workflows both more efficient and sustainable. We'll look into strategies such as canceling redundant jobs, setting appropriate workflow timeouts, optimizing caching techniques, and managing workflow triggers more effectively. ## Cancel Redundant Jobs When active development branches receive rapid updates, whether through direct pushes or pull requests, they often trigger multiple instances of the same workflow. This redundancy can lead to unnecessary resource consumption and potential delays in integration processes. One effective solution is to utilize [concurrency groups](https://docs.github.com/en/enterprise-cloud@latest/actions/using-jobs/using-concurrency#using-concurrency-in-different-scenarios). By assigning a concurrency key, you can group workflows, thereby controlling their execution more effectively. GitHub ensures that within a concurrency group, only one job or workflow runs at a time, limiting it to one running and one pending job. Any new jobs that are triggered at that point will instead be queued. However, by setting `cancel-in-progress` to true, we cancel any ongoing workflows within the same concurrency group when a new one is triggered. This approach effectively clears the queue and ensures that any running workflows are processing the latest changes: ```yaml on: push: branches: - main concurrency: group: integration-tests cancel-in-progress: true ``` For more granular control, particularly useful in environments that utilize branching strategies, consider using dynamic expressions in the group naming. This configuration allows workflows triggered on different branches to be grouped separately, ensuring that updates on one branch do not interfere with the progress of workflows on another: ```yaml concurrency: group: integration-tests-${{ github.ref }} cancel-in-progress: true ``` Employing this strategy optimizes the use of CI resources by avoiding redundant or outdated jobs while maintaining the integrity of our codebase across all types of updates. ## Set Workflow Timeouts The default workflow timeout is 6 hours, which is often excessive for processes that complete within minutes. Setting a more appropriate duration prevents wasteful use of resources, especially in cases where a process might hang due to issues like deadlocks, resource starvation, or software bugs. For instance, a lint check that typically completes within two minutes might only need a 10-minute timeout: ```yaml jobs: lint-checks: runs-on: ubuntu-latest timeout-minutes: 10 ``` This simple adjustment prevents resources from being idly consumed, allowing them to be used for more critical tasks and thereby enhancing the sustainability of your workflows. ## Utilize Caching for Efficiency ### Optimize Dependency Handling Workflow runs often share the exact same dependencies, which must be downloaded each time. Efficient caching can save both time and resources. I recommend using the [setup actions](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#about-caching-workflow-dependencies) for setting up environments like Node.js with caching enabled: ```yaml - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: 20 cache: 'npm' ``` This configuration caches the `~/.npm` directory, which stores tarballs and metadata rather than the `node_modules`, reducing size and enhancing reusability across different environments. Using `npm ci` instead of `npm install` further accelerates builds by installing exact versions from the `package-lock.json`, ensuring reproducible builds and eliminating dependency resolution delays. ### Cache and Restore Build Assets For assets that are resource-intensive to recreate and don't frequently change, consider using the [cache action](https://github.com/actions/cache): ```yaml jobs: build: runs-on: ubuntu-latest steps: - name: Cache Assets uses: actions/cache@v4 with: path: assets key: assets-${{ github.run_id }} restore-keys: | assets- ``` This configuration caches the `assets` directory, restoring it at the start of each build. By referencing the run ID in the cache key, each build attempt will initially seek its own unique cache. If there isn’t an exact match, the `restore-keys` option allows the build to use the most recent cache starting with `assets-`. This technique is particularly useful for processes like generating responsive images, where restoring them from the cache can save significant time and resources. Note: GitHub retains caches for 7 days after the last access. For infrequent workflows, a storage-based solution might be more appropriate. Cache management, including invalidation, can be handled through the repository's [web interface](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#deleting-cache-entries). ## Optimize Workflow Triggers Efficiently using `paths` and `paths-ignore` to specify which files or directories should trigger workflows can help focus resources on meaningful changes. For instance, to exclude documentation updates from triggering workflows: ```yaml on: push: paths-ignore: - 'docs/**' ``` This configuration prevents any change in the `docs` directory from triggering the workflow, conserving resources for non-code updates. To trigger workflows only for changes in front-end code (e.g., JavaScript files): ```yaml on: push: paths: - 'frontend/**.js' ``` This setting ensures that workflows are triggered only when `js` files in the `frontend` directory are modified, ensuring relevant tasks like linting are executed only when necessary. For more advanced usage, refer to the official GitHub Actions documentation [here](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore). Feel free to also check out the [paths-filter](https://github.com/dorny/paths-filter) action which offers more advanced features. ## Leverage ARM Architecture for Efficiency GitHub recently announced [ARM-based runners](https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/) for GitHub Team and Enterprise Cloud plans, with plans to offer them for open-source projects before the year's end. These runners provide improvements in speed and energy consumption due to their reduced instruction set. They consume 30-40% less energy for some of the most widely deployed workflows and are 37% cheaper than their x64 counterparts. However, not all workflows are ideal for ARM runners, especially those that depend on software or libraries not yet optimized for ARM architecture, or that rely on x64-specific optimizations. Testing and evaluating your workflows will help you identify any compatibility issues and measure the potential benefits in terms of speed, cost, and energy consumption. To utilize these ARM runners in your workflows, you first need to create an ARM-based runner as shown in [this video](https://www.youtube.com/watch?v=cgI6SBP8pEM) or documented [here](https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/about-larger-runners). Once set up, you can specify the runner by its name in the `runs-on` field in your GitHub Actions workflow. For example: ```yaml jobs: build: runs-on: my-org-arm-runner ``` This configuration directs GitHub Actions to use the specified ARM-based runner, allowing you to take advantage of the efficiency and cost savings offered by ARM architecture. ## Estimate Workflow Energy Consumption Determining which strategy consumes fewer resources can sometimes be challenging. Tools like the [eco-ci-energy-estimation](https://github.com/green-coding-solutions/eco-ci-energy-estimation) action allow us to get estimates and help us make informed decisions. For instance, to determine whether caching npm dependencies would reduce resource consumption, I ran 5 workflow runs with caching and 5 without. The result showed that caching npm dependencies reduced overall energy consumption and time. ![Energy comparison of using caching vs not using caching](/uploads/workflow-energy-comparison-chart.gif) _Energy comparison of using caching vs not using caching_ The following example demonstrates how to integrate this measurement tool into a workflow. Call `start-measurement` before running the work to be measured, and `get-measurement` and `display-results` after. ```yaml - name: Initialize Energy Estimation uses: green-coding-solutions/eco-ci-energy-estimation@v3 with: task: start-measurement - name: Tests measurement uses: green-coding-solutions/eco-ci-energy-estimation@v3 with: task: get-measurement label: 'Install' - name: Show Energy Results uses: green-coding-solutions/eco-ci-energy-estimation@v3 with: task: display-results ``` Upon completion, the workflow produces a summary that looks like this: ![Workflow summary showing measurement estimates](/uploads/workflow-energy-estimation-summary.gif) _Workflow summary showing measurement estimates_ The action sends metrics data to metrics.green-coding.io by default, including energy values, duration of measurements, CPU model, and details about your repository and commit. This data is used to create a badge and a detailed display of your CI runs’ energy consumption, viewable [here](https://metrics.green-coding.io/ci-index.html). If you prefer not to share this data, simply set `send-data` to `false` in the action’s configuration. ## More Ways to Optimize Workflows With these simple tips, you can significantly reduce costs, save time, and better allocate resources. For more tips, I highly recommend the presentation "[Things your Pipeline Should (Not) Do](https://www.youtube.com/watch?v=mYBkSg1dz2Y)" by my colleague Raniz, who inspired me to write this post. --- # Supply Chain Security Risks in Static Sites URL: https://nizar.se/supply-chain-security-risks-in-static-sites/ Date: 2024-09-11 Static sites are perceived as more secure than their dynamic counterparts due to their simplicity and lack of server-side components. However, this perception can lead to overlooked threats, particularly those introduced by third-party libraries and services. In this article, we'll explore how these vulnerabilities arise and how you can defend your site against them. Static sites are perceived as more secure than their dynamic counterparts due to their simplicity and lack of server-side components. However, this perception can lead to overlooked threats, particularly those introduced by third-party libraries and services. In this article, we'll explore how these vulnerabilities arise and how you can defend your site against them. ## What Are Static Sites? Static sites consist of pre-rendered HTML, CSS, and JavaScript files that are delivered directly to the user's browser. Their appeal lies in their speed, scalability, and cost-effectiveness. By avoiding server-side processing, static sites improve performance and reduce resource consumption. However, many static sites rely on third-party JavaScript libraries or external services for added functionality, such as form handling or analytics. This introduces supply chain vulnerabilities, where compromised dependencies become attack vectors. ## The Threat of Supply Chain Attacks Supply chain attacks occur when third-party code that your site relies on is compromised. One notorious example is the [Polyfill.io attack](https://sansec.io/research/polyfill-supply-chain-attack), where malicious code affected over 100,000 websites. The code redirected users to a sports betting site while evading detection: > The code has specific protection against reverse engineering, and only activates on specific > mobile devices at specific hours. It also does not activate when it detects an admin user. It also > delays execution when a web analytics service is found, presumably to not end up in the stats. In another incident, Cloudflare's CDNJS service exposed up to [12% of all websites](https://www.bleepingcomputer.com/news/security/critical-cloudflare-cdn-flaw-allowed-compromise-of-12-percent-of-all-sites/) on the internet to potential code injection attacks, showing that even well-regarded Content Delivery Networks (CDNs) can be vulnerable. ## Defending Against Supply Chain Attacks Key defenses against supply chain attacks include [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) (SRI) and [Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) (CSP). These tools work best when used together as part of a layered security approach. ### Subresource Integrity (SRI) Subresource Integrity (SRI) ensures that fetched resources, such as JavaScript files, haven't been tampered with. It does so by comparing the resource's cryptographic hash to a predefined value. If there's a mismatch, the browser blocks the resource from loading, preventing potential malicious code from executing. For example: ```html ``` In this example, the `integrity` attribute includes the hash of the file. This ensures the resource is unaltered, adding an extra layer of protection for external scripts your site relies on. ### Content Security Policy (CSP) Content Security Policy (CSP) allows you to define which sources are permitted to load resources such as scripts, styles, and images. This reduces the risk of code injection by blocking unauthorized content from running on your site. Here is an example of a CSP that restricts script loading to the same domain and a trusted CDN: ```text Content-Security-Policy: script-src 'self' https://cdn.example.com; ``` You can further strengthen your policy with hash-based CSP directives, allowing only scripts with matching cryptographic hashes to execute, regardless of their source: ```text Content-Security-Policy: script-src 'sha384-abc123...' 'sha384-def456...'; ``` This approach ensures that unauthorized scripts won't run even if the source is compromised. ### Addressing Security Threats Using both CSP and SRI together enhances security, as CSP controls where resources can be loaded from while SRI verifies their integrity. According to the [OWASP CSP Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Content_Security_Policy_Cheat_Sheet.html#defense-in-depth): > Even on a fully static website, which does not accept any user input, a CSP can be used to enforce > the use of Subresource Integrity (SRI). This can help prevent malicious code from being loaded on > the website if one of the third-party sites hosting JavaScript files (such as analytics scripts) > is compromised. CSP and SRI help mitigate common security threats such as: - **Cross-Site Scripting (XSS)**: XSS attacks allow malicious scripts to be injected into your site, potentially leading to data theft, session hijacking, or unauthorized actions. CSP blocks unauthorized scripts by restricting which can run. - **Clickjacking**: This attack deceives users into interacting with hidden elements, often resulting in unintended actions like form submissions or transactions. CSP can prevent clickjacking by blocking your site from being embedded in iframes, a common method used in such attacks. - **Site Defacement**: Attackers may try to alter content on your website, damaging its credibility. A combination of CSP and SRI ensures that only authorized content is loaded, significantly reducing the risk of defacement. ### Best Practices for CSP Implementation Despite CSP's effectiveness, it remains underutilized. A [2020 survey](https://www.rapid7.com/blog/post/2020/11/02/overview-of-content-security-policies-csp-on-the-web/) of the Alexa Top 1 Million websites found that only 7% had a valid CSP, and a [Bitsight study](https://www.bitsight.com/blog/content-security-policy-limits-dangerous-activity-so-why-isnt-everyone-doing-it) revealed only 2% of 5 million web applications were fully secure. To fully leverage CSP, follow these best practices: - **Use a strict default policy**: Start with `default-src 'none'` and then explicitly allow trusted sources. - **Apply restrictions to all sources**: Use `style-src`, `media-src`, and other directives to tightly control which external resources are loaded. - **Avoid unsafe directives**: Directives like `unsafe-inline` and `unsafe-eval` can undermine your policy by allowing inline scripts and styles. - **Prevent clickjacking**: Use `frame-ancestors` to block other sites from embedding your content in iframes. - **Test with reporting**: Before enforcing CSP, use `report-only` mode to monitor violations without disrupting functionality. ## Automating Security Practices Managing CSP manually can be error-prone, so automation tools are invaluable for maintaining consistent enforcement without burdening your workflow. At [factor10](https://www.factor10.com/), we automate several processes to reduce human error while ensuring flexibility for quick interventions: 1. **Controlled Script Updates**: During our build process, a utility fetches the latest versions of third-party scripts, but changes are only introduced after a manual diff-check, giving us full control over updates. 2. **Automated SRI Generation**: Hashes for all fetched scripts are automatically calculated and embedded into the HTML, ensuring integrity. 3. **Automated Security Headers**: CSP rules are generated and applied to our NGINX configuration during deployment. If any verification fails, the deployment is discarded, maintaining the security of the live site. By automating these steps and pairing them with monitoring tools, we ensure consistent security practices. Alerts notify our team of any deviations, allowing us to maintain control and respond promptly. ## Tools for Automating and Testing Security The following tools can help you automate security practices and assess or improve your site's security posture: - [Google’s CSP Evaluator](https://csp-evaluator.withgoogle.com/): Analyzes your CSP and offers suggestions for strengthening it. - [Mozilla Observatory](https://observatory.mozilla.org/): Provides insights into security headers and suggests improvements. - [securityheaders.com](https://securityheaders.com/): Tests and rates your site's HTTP headers based on best practices. - [Playwright](https://playwright.dev/): A versatile tool for cross-browser testing to ensure consistent security behavior. - [Webperf.se](https://webperf.se): Offers an [open-source suite](https://github.com/Webperf-se/webperf_core) for analyzing performance, accessibility, and security. For our [sustainable-websites](https://factor10.com/websites/), we use [webperf's premium service](https://webperf.se/erbjudande/) for daily tests, alongside automated alerts via [webhooks](https://webperf.se/articles/webhooks/) on Slack to notify us of any issues. ## Conclusion While static websites are simpler than dynamic ones, they are not immune to modern supply chain threats. The Polyfill.io incident serves as a reminder that even static sites can be compromised by third-party dependencies. Implementing security measures like Subresource Integrity (SRI) and Content Security Policy (CSP), combined with automation, significantly reduces your site's exposure to supply chain risks. However, no defense strategy is complete without regular testing and monitoring. Vigilance, combined with automation, ensures that your site remains resilient in the face of evolving threats without sacrificing performance or efficiency. --- # Implementing Durable Entities in Kotlin URL: https://nizar.se/implementing-durable-entities-in-kotlin/ Date: 2024-11-19 This series explores how to implement Durable Entities in Kotlin, which are stateful objects that manage their own state and enable seamless persistence and resumption in serverless workflows. Though not natively supported in Kotlin, this series provides a step-by-step guide to replicating their functionality using orchestration instances. The series covers topics such as managing state, signaling entities, and cross-entity communication. I've been fortunate to work with some of the [greatest minds](https://factor10.com/) early in my career, an opportunity I'm endlessly grateful for. Over the past year, I've worked with Erik Meijer on [Automind](https://fortune.com/2024/04/02/mark-zuckerberg-ai-jobs-meta-brain-drain-erik-meijer/), tackling some truly exciting challenges. This experience introduced me to Durable Functions, an abstraction that simplifies building stateful workflows in serverless environments. Among its many features, Durable Entities stands out by enabling developers to model stateful objects that manage their own state. However, there's a catch: Durable Entities aren't supported in Kotlin or Java. Knowing Erik, suggesting another language was out of the question. This set the stage for my challenge: implementing Durable Entities in Kotlin. ## What are Durable Entities? Before diving into implementation, let's first understand Durable Entities. They enable applications to persist state across executions, conserve resources when idle, and seamlessly resume workflows when needed. This makes them particularly valuable in scenarios like: - Event-driven architectures. - Interaction-based workflows. - Distributed workflows. In essence, Durable Entities behave like long-lived, lightweight actors. They preserve their state using event sourcing, a pattern where state changes are logged as events, allowing state to be reconstructed by replaying these events. Entities also execute operations deterministically, ensuring consistency even in distributed environments. For example, the official C# documentation provides a simple counter entity, which maintains a value and supports operations like adding, resetting, and retrieving the current value: ```csharp [FunctionName("Counter")] public static void Counter([EntityTrigger] IDurableEntityContext ctx) { ctx.OperationName.ToLowerInvariant() switch { "add" => ctx.SetState(ctx.GetState() + ctx.GetInput()), "reset" => ctx.SetState(0), "get" => ctx.Return(ctx.GetState()) }; } ``` This functionality is powerful, but replicating it in Kotlin requires a creative approach. Let's dive into how that can be achieved. ## The Challenge and Insight Initially, I explored gRPC and the [durabletask-protobuf](https://github.com/microsoft/durabletask-protobuf) library, inspired by implementations in other languages. While this approach was promising, it wasn't successful. A critical insight came from [Chris Gillum](https://github.com/cgillum), the creator of Durable Functions, who highlighted that Durable Entities are [implemented in other languages using orchestration instances](https://github.com/microsoft/durabletask-java/issues/194#issuecomment-2397984973). This realization was a turning point, and not long after, I had my first Durable Entity in Kotlin. ## Introducing the Series This series will guide you through the development process, step by step. Along the way, we'll examine the principle and constraints that underpin Durable Entities, ensuring that our implementation stays true to the design requirements. Here's an example of a counter entity implemented in Kotlin, modeled after the C# example above. It demonstrates how to define an entity and handle operations like "add", "reset", and "get": ```kotlin @FunctionName("Counter") fun counter(@DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext) { asEntity(ctx, initialState = 0) { when (operationName.lowercase()) { "add" -> setState(getState() + getInput()) "reset" -> setState(0) "get" -> returnResult(getState()) } } } ``` ## What's Next This series will cover the following topics step-by-step: 1. [Persisting State](/implementing-durable-entities-in-kotlin-persisting-state/) 2. [Creating Entities](/implementing-durable-entities-in-kotlin-creating-entities/) 3. [Signaling Entities](/implementing-durable-entities-in-kotlin-signaling-entities/) 4. [Signaling Entities with Input](/implementing-durable-entities-in-kotlin-signaling-entities-with-input/) 5. [Cross-Communication](/implementing-durable-entities-in-kotlin-cross-communication/) 6. [Calling Entities](/implementing-durable-entities-in-kotlin-calling-entities/) 7. [Final Enhancements](/implementing-durable-entities-in-kotlin-final-enhancements/) Whether you're already familiar with Durable Functions or exploring stateful serverless workflows for the first time, this series will offer something for everyone. Stay tuned! --- # Implementing Durable Entities in Kotlin - Persisting State URL: https://nizar.se/implementing-durable-entities-in-kotlin-persisting-state/ Date: 2025-01-01 Durable Entities require a structured way to reference specific instances. This post introduces Entity IDs, which uniquely identify entity instances, and refines entity creation with a lightweight abstraction. By aligning with how Durable Entities are structured in other languages, we make instance creation more intuitive while keeping the implementation clean and flexible. ## Introduction Durable Entities provide a way to model long-lived, stateful objects in serverless workflows, but they aren't natively supported in Kotlin. In the [previous post](/implementing-durable-entities-in-kotlin/), we introduced Durable Entities and explored their key principles, such as event sourcing and deterministic execution. A fundamental requirement of Durable Entities is persisting state across executions to ensure entities remain functional between invocations. Since Durable Entities aren't available in Kotlin, we can achieve this using custom status metadata, a built-in feature of Durable Orchestrations that allows storing arbitrary state tied to an instance. This post demonstrates how to persist and retrieve state using custom status. We will walk through initializing and reading the state of a Counter entity, establishing a foundation for more advanced features in later posts. ## Persisting State in an Orchestration The first step in making an entity stateful is to ensure it can store and retain values between executions. We define a Counter orchestration that initializes a stored value. ```kotlin @FunctionName("Counter") fun counter(@DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext) { // Set initial value ctx.setCustomStatus(0) } ``` This function initializes a Counter entity and assigns it a starting value of 0 using `ctx.setCustomStatus(0)`. The orchestration is currently static as it stores a value but does not yet support modifications to it. In later posts, we will extend it to allow updates through external inputs. ## Creating An Orchestration Instance Before interacting with the Counter, we need to create instances of it. In Durable Functions, orchestrations are typically instantiated using HTTP-triggered functions, which allow external systems to create and manage them on demand. The following function starts a new instance of the Counter orchestration: ```kotlin @FunctionName("Create") fun create( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { // Start a new Counter instance val instanceId = ctx.client.scheduleNewOrchestrationInstance("Counter") // Return the instance ID so we can interact with it later return request .createResponseBuilder(HttpStatus.OK) .body("Created Counter with ID: $instanceId\n") .build() } ``` The function starts an orchestration by calling `scheduleNewOrchestrationInstance("Counter")` and returns the generated instance ID which will be used later to interact with it. We can now create a counter instance using `curl`: ```shell ❯ curl -s https://durable-app.azurewebsites.net/api/Create Created Counter with ID: 9864a3e8-4f01-4201-8920-0b628c739bee ``` At this point, we have an instance that persists state, but we need a way to retrieve its stored value. ## Retrieving State from the Orchestration To fetch the stored value of the Counter, we define another HTTP-triggered function: ```kotlin @FunctionName("Get") fun get( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { // Extract instanceId from query parameters val instanceId = request.queryParameters["instanceId"] ?: return request.createResponseBuilder(HttpStatus.BAD_REQUEST) .body("instanceId required\n") .build() // Retrieve the counter value from custom status val counterValue = ctx.client .getInstanceMetadata(instanceId, true) ?.readCustomStatusAs(Int::class.java) // Return the current counter value return request.createResponseBuilder(HttpStatus.OK) .body("Counter value: $counterValue\n") .build() } ``` This function looks up an orchestration instance by its `instanceId`, retrieves its stored custom status, and returns the current counter value. To make the retrieval logic more reusable and maintainable, we introduce an extension function: ```kotlin fun DurableTaskClient.getEntityState(entityId: String, returnType: Class): T? { return this.getInstanceMetadata(entityId, true)?.readCustomStatusAs(returnType) } ``` With this abstraction, our `Get` function becomes cleaner and more intuitive: ```kotlin @FunctionName("Get") fun get( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { // Get entityId from request query parameters val entity = request.queryParameters["entityId"] ?: return request.badRequest("entityId required") // Get the state of the Counter entity as an Int val counterValue = ctx.client.getEntityState(entity, Int::class.java) // Return an HTTP response containing counter's current value return request.createResponseBuilder(HttpStatus.OK) .body("Counter value: $counterValue\n") .build() } ``` This approach hides implementation details, keeping our code readable. ## Verifying State Persistence Now that we can create and retrieve a counter instance, let us verify that the state persists: ```shell ❯ curl -s https://durable-app.azurewebsites.net/api/Create Created Counter with ID: af842648-5ffe-4be3-bb17-27aabf303dcb ❯ curl -s https://durable-app.azurewebsites.net/api/Get\?\ entityId\=af842648-5ffe-4be3-bb17-27aabf303dcb Counter value: 0 ``` The counter instance correctly initializes and maintains its state. Even if we rerun the second command hours or days later, it will still return the same value. This persistence will become more evident when we introduce event-driven updates in later posts. ## What's Next? Now that our Counter entity can persists state, the next step is refine its structure. Before making it interactive, we need a better way to reference and manage instances. In the next post, we introduce Entity IDs, which provide a structured way to uniquely identify entities while also abstracting orchestration scheduling. ## Read Further This post is part of the [Implementing Durable Entities in Kotlin](/implementing-durable-entities-in-kotlin) series. - **Previous**: [Series Introduction](/implementing-durable-entities-in-kotlin). - **Next**: [Creating Entities](/implementing-durable-entities-in-kotlin-creating-entities). --- # Implementing Durable Entities in Kotlin - Creating Entities URL: https://nizar.se/implementing-durable-entities-in-kotlin-creating-entities/ Date: 2025-01-02 So far, we've learned how Durable Entities can persist state across executions and conserve resources, and we've used custom status and metadata to manage state for a basic counter entity. In this post, we'll build on that by introducing Entity IDs which are unique identifiers that allow us to create and interact with specific entities. By abstracting entity creation into a reusable pattern, we simplify the process and prepare for handling operations efficiently in upcoming posts. ## Introduction So far, we have seen how Durable Entities persist state across executions and conserve resources. We have used custom status and metadata to store and retrieve state for a basic Counter entity. However, to interact with entities efficiently, we need a structured way to identify and reference them. This post introduce Entity IDs, which uniquely identify specific instances of an entity. To make instance creation simpler and more intuitive, we also introduce a lightweight abstraction that aligns with how Durable Entities are created in other languages. ## Understanding Entity IDs When working with Durable Entities, each instance must be uniquely identifiable. According to the [Durable Entities documentation](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-entities?tabs=function-based%2Cin-process%2Cpython-v2&pivots=csharp#entity-id), an entity ID consists of an entity name, which represents the type of the entity, and an entity key, which uniquely identifies a specific instance. This structure enables the creation of multiple instances of the same entity while keeping them distinct. For example, in C#, an entity ID for a counter entity can be created like this: ```csharp var entityId = new EntityId(nameof(Counter), "myCounter"); ``` Here, `Counter` is the entity name, and `myCounter` is the key that uniquely identifies this instance. With this approach, we can reference and interact with specific entities in a structured way. ## Abstracting Entity Creation ### Why introduce an Abstraction? Right now, we manually schedule an orchestration each time we create an entity. While this approach works, it exposes unnecessary implementation details. Instead of handling entity creating explicitly in each function, we can introduce an abstraction that simplifies the process. Encapsulating entity creation in a dedicated class makes instance creation more intuitive, aligns with Durable Entities in other languages, and keeps the code cleaner. ### Implementing the Entity Class To achieve this, we define an `Entity` class that takes an entity name, and optional key to distinguish instances, and an optional input parameter. It uses DurableTaskClient to schedule an orchestration and returns the instance ID. ```kotlin class Entity private constructor( client: DurableTaskClient, name: String, key: String? = null, input: Any? = null, ) { // Create an instance of the requested Entity val instanceId: String = client.scheduleNewOrchestrationInstance(name, input, key) // Factory method for entity creation companion object { operator fun invoke( client: DurableTaskClient, name: String, input: Any? = null, key: String? = null, ): String { val entity = Entity(client, name, input, key) return entity.instanceId } } } ``` With this abstraction in place, creating an entity instance is now as simple as: ```kotlin val entityId = Entity(ctx.client, "Counter", "myCounter") ``` At this stage, we return the instance ID, but we could modify it to return a formatted string like `@Counter@Game1` to align more closely with the native implementation. Note that we have also added an input parameter as well. We will use this later when we want to provide input to an entity when we instantiate it. ## Refactoring the Create Function With the new abstraction, we can simplify our `Create` function: ```kotlin @FunctionName("Create") fun create( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { // Create a new Counter instance with a specific key val entityId = Entity(ctx.client, "Counter", "myCounter") // Return an HTTP response containing the entity ID return request .createResponseBuilder(HttpStatus.OK) .body("Created Counter with ID: $entityId\n") .build() } ``` This keeps the code focused on the business logic rather than exposing orchestration details. ## Simplifying HTTP Handling Since we will be handling multiple HTTP-triggered functions, we can reduce repetitive response-handling code by introducing two extension functions: ```kotlin fun HttpRequestMessage>.badRequest( message: String ): HttpResponseMessage = createResponseBuilder(HttpStatus.BAD_REQUEST).body(message).build() fun HttpRequestMessage>.success(message: String): HttpResponseMessage = createResponseBuilder(HttpStatus.OK).body(message).build() ``` Using these, our `Create` function becomes even cleaner: ```kotlin @FunctionName("Create") fun create( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { val entityId = Entity(ctx.client, "Counter", "myCounter") return request.success("Created Counter with ID: $entityId\n") } ``` This reduces clutter and makes the functions easier to read. ## What's Next? With Entity IDs in place, we now have a structured way to create and reference Durable Entities. The next step is to make these entities interactive by introducing event-driven updates. In the next post, we will explore signaling, a mechanism that allows sending operations to entities without awaiting a response. ## Read Further This post is part of the [Implementing Durable Entities in Kotlin](/implementing-durable-entities-in-kotlin) series. - **Previous**: [Persisting State](/implementing-durable-entities-in-kotlin-persisting-state). - **Next**: [Signaling Entities](/implementing-durable-entities-in-kotlin-signaling-entities). --- # Implementing Durable Entities in Kotlin - Signaling Entities URL: https://nizar.se/implementing-durable-entities-in-kotlin-signaling-entities/ Date: 2025-01-03 Durable Entities remain active across executions, but without external input, they are static. This post introduces signaling, a one-way communication method that enables entities to react to events dynamically. Using a Counter example, we extend our entity to listen for increment and reset operations, making it interactive. By the end, our counter will respond to external requests, laying the foundation for more advanced event-driven workflows. ## Introduction So far, we have created an entity that stores state using custom status, but it remains static. Once created, there is no way to interact with it or modify its state dynamically. In real-world applications, stateful objects must react to external inputs. This is where event-driven updates come in. In this post, we introduce signaling, a mechanism of sending one-way (fire-and-forget) messages to an entity. By enabling our counter to listen for external events, we allow it to dynamically update its state in response to operations like increment and reset. ## Signaling vs. Calling Entities can be accessed in two ways, as explained in the [Durable Entities documentation](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-entities?tabs=function-based%2Cin-process%2Cpython-v2&pivots=csharp#access-entities): - **Signaling** Sends an operation without waiting for a response. It is best suited for updates that do not require immediate feedback. - **Calling** Sends an operation and waits for a result before continuing. This is useful when you need to retrieve values from an entity. In this post, we will focus on signaling and by the end of it our counter entity will become interactive, responding to external requests. ## Extending the Counter to Handle Events To enable event-driven updates, we modify the `Counter` orchestration so that it listens for incoming events. It will wait for external signals and update its state accordingly. ```kotlin @FunctionName("Counter") fun counter(@DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext) { // Initialize the counter value and the custom status var counter = 0 ctx.setCustomStatus(counter) while (true) { // Wait for an external event val operation = ctx.waitForExternalEvent("EntityOperation", String::class.java).await() // Process the requested operation when (operation) { "Increment" -> counter++ // Increase counter by 1 "Reset" -> counter = 0 // Reset counter to 0 } // Update the stored state with the new value ctx.setCustomStatus(counter) } } ``` Now, instead of being static, the counter remains active, continuously listening for events. When it receives an operation, it updates its state and waits for the next event. At this point, the counter can respond to external events, but we still need a way to send these signals. ## Sending Signals to the Counter To interact with the counter, we create HTTP-triggered functions that send events to a specific instance. ### Increment Operation This function sends an event to increase the counter value. ```kotlin @FunctionName("Increment") fun increment( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { // Extract the instanceId from query parameters val entityId = request.queryParameters["entityId"] ?: return request.badRequest("entityId required") // Raise an event to the Counter to increment its value ctx.client.raiseEvent(instanceId, "EntityOperation", "Increment") // Return an HTTP response with a helpful message return request.success("Counter Incremented.") } ``` ### Reset Operation This function sends an event to reset the counter to 0. ```kotlin @FunctionName("Reset") fun increment( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { // Extract the instanceId from query parameters val entityId = request.queryParameters["entityId"] ?: return request.badRequest("entityId required") // Raise an event to the Counter to reset its value ctx.client.raiseEvent(instanceId, "EntityOperation", "Reset") // Return an HTTP response with a helpful message return request.success("Counter Reset.") } ``` ### Abstracting Signal Logic Right now, we hardcoded the event name `EntityOperation` everywhere. This is an implementation detail and there is no reason to leak into the rest of the code base. We can introduce a helper function for signaling that will help us clean up our code: ```kotlin fun DurableTaskClient.signalEntity(entityId: String, operation: String) { this.raiseEvent(entityId, "EntityOperation", operation) } ``` With our abstraction, the HTTP functions become even cleaner: ```kotlin @FunctionName("Increment") fun increment( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { val entityId = request.queryParameters["entityId"] ?: return request.badRequest("entityId required") ctx.client.signalEntity(entityId, "Increment") return request.success("Counter Incremented.") } ``` This makes signaling cleaner and more intuitive, matching how Durable Entities are used in other languages. ## Verifying the Implementation With everything in place, we can now verify that our counter responds to events. ```shell ❯ curl -s https://durable-app.azurewebsites.net/api/Create Created Counter with ID: b13e7165-6ee4-4429-9295-a5f998d9881d ❯ curl -s https://durable-app.azurewebsites.net/api/Get\?\ entityId\=b13e7165-6ee4-4429-9295-a5f998d9881d Counter value: 0 ❯ curl -s https://durable-app.azurewebsites.net/api/Increment\?\ entityId\=b13e7165-6ee4-4429-9295-a5f998d9881d Counter Incremented. ❯ curl -s https://durable-app.azurewebsites.net/api/Get\?\ entityId\=b13e7165-6ee4-4429-9295-a5f998d9881d Counter value: 1 ❯ curl -s https://durable-app.azurewebsites.net/api/Reset\?\ entityId\=b13e7165-6ee4-4429-9295-a5f998d9881d Counter Reset. ❯ curl -s https://durable-app.azurewebsites.net/api/Get\?\ entityId\=b13e7165-6ee4-4429-9295-a5f998d9881d Counter value: 0 ``` With this, we have successfully made our counter interactive. It now listens for external events and updates its state dynamically. ## What's Next? Now that we can send one-way messages to update an entity, the next step is to make these signals more powerful. In the next post, we will extend signaling to support input parameters, allowing entities to process structured commands rather than just operation names. ## Read Further This post is part of the [Implementing Durable Entities in Kotlin](/implementing-durable-entities-in-kotlin) series. - **Previous**: [Creating Entities](/implementing-durable-entities-in-kotlin-creating-entities). - **Next**: [Signaling Entities with Input](/implementing-durable-entities-in-kotlin-signaling-entities-with-input). --- # Implementing Durable Entities in Kotlin - Signaling Entities with Input URL: https://nizar.se/implementing-durable-entities-in-kotlin-signaling-entities-with-input/ Date: 2025-01-04 Signaling allows entities to process operations dynamically, but until now, signals have been limited to simple operation names. In this post, we extend our implementation to support input data, enabling entities to handle structured commands. By passing values with signals, our counter can now process operations like adding a specified amount. This enhancement brings our entities closer to real-world applications where interactions require more than basic state changes. ## Introduction So far, we have implemented signaling to interact with entities dynamically, allowing our counter to respond to operations like increment and reset. However, these signals have been limited to simple operation names, restricting flexibility. In this post, we extend our signaling mechanism to support input data, enabling more complex interactions. By passing structured data with each signal, we allow operations like adding a specific amount to the counter. With this enhancement, our counter entity will support more realistic interactions, moving closer to the flexibility required in real-world applications. ## Passing Input with Signals ### Defining a Data Model for Operations Instead of sending plain operation names, we define a structured `Operation` data class that includes both an operation type and optional input. Since Durable Functions use Jackson for serialization, we follow the same approach for consistency. ```kotlin val objectMapper = jacksonObjectMapper() data class Operation( val name: String, val input: Any? = null ) ``` This allows signals to carry additional information, such as an integer to modify the counter. ### Updating the SignalEntity Function To handle structured input, we modify the `signalEntity` function to send JSON-encoded operations. ```kotlin fun DurableTaskClient.signalEntity( entityId: String, operationName: String, operationInput: Any? = null ) { val operation = Operation(name = operationName, input = operationInput) val operationJSON = objectMapper.writeValueAsString(operation) this.raiseEvent(entityId, "EntityOperation", operationJSON) } ``` This ensures that every entity receives structured events, making event-driven interactions more powerful. ## Handling Input in The Counter Entity ### Processing Events with Data Input We modify the `Counter` entity to parse structured signals and apply operations accordingly. ```kotlin @FunctionName("Counter") fun counter(@DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext) { var counter = 0 ctx.setCustomStatus(counter) while (true) { val operationJson = ctx.waitForExternalEvent("EntityOperation", String::class.java).await() val operation = objectMapper.readValue(operationJson) when (operation.name) { "Increment" -> counter++ "Add" -> counter += (operation.input as? Int) ?: 0 "Reset" -> counter = 0 } ctx.setCustomStatus(counter) } } ``` Now, the counter can increment by one, add a custom value, or reset based on structured input. ## Sending Input from an HTTP Function To take advantage of this functionality, we introduce an HTTP function that allows users to specify an amount to be added to the counter. ```kotlin @FunctionName("Add") fun add( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { val entityId = request.queryParameters["entityId"] ?: return request.badRequest("entityId required") val amount = request.queryParameters["amount"] ?: return request.badRequest("amount required") ctx.client.signalEntity(entityId, "Add", amount.toInt()) return request.success("Amount added.") } ``` This function extracts the `entityId` and `amount` from the request, signals the entity with the provided value, and finally returns a success message. ## Testing the Implementation With everything in place, we can now verify that the `Add` function correctly updates the counter. ```shell ❯ curl -s https://durable-app.azurewebsites.net/api/Create Created Counter with ID: 42587159-ca0e-4342-8514-2cdeee214953 ❯ curl -s https://durable-app.azurewebsites.net/api/Add\?\ entityId\=42587159-ca0e-4342-8514-2cdeee214953\&amount\=7 Amount added. ❯ curl -s https://durable-app.azurewebsites.net/api/Get\?\ entityId\=42587159-ca0e-4342-8514-2cdeee214953 Counter value: 7 ❯ curl -s https://durable-app.azurewebsites.net/api/Add\?\ entityId\=42587159-ca0e-4342-8514-2cdeee214953\&amount\=2 Amount added. ❯ curl -s https://durable-app.azurewebsites.net/api/Get\?\ entityId\=42587159-ca0e-4342-8514-2cdeee214953 Counter value: 9 ``` The counter correctly updates based on the values provided. ## Refactoring for Cleaner Event Handling ### Simplifying Event Handling with a Helper Function To streamline the counter's logic, we introduce a helper function that waits for and parses incoming operations: ```kotlin fun TaskOrchestrationContext.waitForEntityOperation(): Operation { val json = this.waitForExternalEvent("EntityOperation", String::class.java).await() return objectMapper.readValue(json) } ``` ### Refactoring the Counter Entity Using this helper function, we simplify our counter implementation. ```kotlin @FunctionName("Counter") fun counter(@DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext) { var counter = 0 ctx.setCustomStatus(counter) while (true) { val operation = ctx.waitForEntityOperation() when (operation.name) { "Add" -> counter += (operation.input as? Int) ?: 0 "Reset" -> counter = 0 } ctx.setCustomStatus(counter) } } ``` This update eliminates unnecessary JSON parsing within the main loop, keeping the entity concise and readable. ## What's Next? Now that our entities can process structured signals, they are no longer limited to basic state changes but can dynamically react to user input. In the next post, we will explore cross-entity communication, enabling entities to interact with each other, further expanding the scope of what we can build. ## Read Further This post is part of the [Implementing Durable Entities in Kotlin](/implementing-durable-entities-in-kotlin) series. - **Previous**: [Signaling Entities](/implementing-durable-entities-in-kotlin-signaling-entities). - **Next**: [Cross-Communication](/implementing-durable-entities-in-kotlin-cross-communication). --- # Implementing Durable Entities in Kotlin - Cross Communication URL: https://nizar.se/implementing-durable-entities-in-kotlin-cross-communication/ Date: 2025-01-05 Entities in a distributed system often need to collaborate, but Durable Functions impose constraints on direct communication. In this post, we introduce cross-entity signaling while maintaining orchestration integrity. By implementing a monitor entity that listens for updates from a counter, we explore structured signaling models, activity functions, and best practices for reliable inter-entity communication in Kotlin. ## Introduction So far, we have built stateful entities that persist data and respond dynamically to external input through signaling. However, all interactions have been one-directional: clients send commands, but entities remain isolated from each other. Real-world applications often require entities to collaborate, exchanging updates and triggering actions across a distributed system. This post introduces cross-entity communication, where entities can signal each other and work together toward a shared goal. To demonstrate this, we introduce a monitor entity that listens for updates from the counter entity whenever it reaches specific milestones. ## Why Can't We Signal Entities Directly? Signaling an entity from another entity or orchestrator using `DurableTaskClient` might seem straightforward, but it introduces reliability issues. Orchestrators replay executions for fault tolerance and scalability. If they interact directly with external clients, they risk sending duplicate signals, leading to unintended side effects. This constraint is fundamental to Durable Functions, ensuring consistency in distributed workflows, as detailed in the [Durable Entities constraints documentation](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-code-constraints?tabs=csharp#bindings). Instead of allowing direct signaling, we introduce an activity function to handle communication. Activities do not replay, ensuring that each signal is sent exactly once while maintaining orchestration integrity. ## Enabling Cross-Entity Communication To ensure structured and reliable entity signaling, we introduce a signaling model that standardizes communication between entities. ### Defining a Structured Signaling Model Instead of sending raw data, we define a structured format for consistent and predictable inter-entity communication. The `Operation` class remains unchanged and represents an action an entity should take, optionally carrying input data. The `SignalEntityInput` class bundles the target entity ID with the operation, making signaling clearer and reusable. ```kotlin data class Operation( val name: String, val input: Any? = null ) data class SignalEntityInput( val entityId: String, val operation: Operation ) ``` By encapsulating signaling details, this model simplifies passing information to the activity function. ## Handling Signals with an Activity Function With a structured model in place, we define an activity function to safely handle signaling. Unlike orchestrations, activities do not replay, making them the correct place to interact with `DurableTaskClient`. ```kotlin @FunctionName("SignalEntity") fun signalEntityActivity( @DurableActivityTrigger(name = "input") input: SignalEntityInput, @DurableClientInput(name = "durableContext") durableContext: DurableClientContext, ) { durableContext.client.signalEntity( input.entityId, input.operation.name, input.operation.input, ) } ``` The activity takes a `SignalEntityInput` and sends a signal on behalf of the caller. Since activities execute only once, they prevent duplicate signals, ensuring reliable inter-entity communication. ## Updating the Counter to Signal the Monitor Now that we have a safe signaling mechanism, we modify the `Counter` entity to notify the `Monitor` when reaching specific milestones. The counter initializes its state, listens for operations, and updates its value accordingly. Whenever it reaches a multiple of 10, it signals the monitor. ```kotlin @FunctionName("Counter") fun counter(@DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext) { var counter = 0 ctx.setCustomStatus(counter) val monitorEntityId = ctx.getInput(String::class.java) while (true) { val operation = ctx.waitForEntityOperation() when (operation.name) { "Add" -> counter += (operation.input as? Int) ?: 0 "Reset" -> counter = 0 } ctx.setCustomStatus(counter) if (counter % 10 == 0) { val responseOperation = Operation(name = "Update", input = counter) val signalInput = SignalEntityInput(monitorEntityId, responseOperation) ctx.callActivity("SignalEntity", signalInput) } } } ``` The counter now receives the monitor's ID as input and signals it when reaching key milestones. Instead of handling the signaling directly, it delegates the task to the activity function, ensuring orchestration constraints are respected. ## Defining the Monitor The `Monitor` entity listens for updates from the `Counter` and logs the received progress. ```kotlin @FunctionName("Monitor") fun monitor( @DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext, context: ExecutionContext, ) { while (true) { val operation = ctx.waitForEntityOperation() when (operation.name) { "Update" -> { context.logger.info("Progress update: ${operation.input}%") } } } } ``` By continuously listening for updates, the monitor keeps track of milestones reached by the counter. ## Initializing Both Entities via HTTP To establish communication between the `Counter` and `Monitor`, we define an HTTP function that creates both entities in a single request. The monitor is created first, and its ID is passed to the counter. ```kotlin @FunctionName("CreateMonitorAndCounter") fun createMonitorAndCounter( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { // Create a Monitor and a Counter, passing the monitorId to the counter val monitorId = Entity(ctx.client, "Monitor") val counterId = Entity(ctx.client, "Counter", monitorId) // Return an HTTP response with the IDs val responseBody = """ Created Monitor and Counter. Monitor ID: $monitorId Counter ID: $counterId """ .trimIndent() return request.success(responseBody) } ``` This ensures the counter knows where to send updates. ## Verifying Cross-Communication We can now verify the `Counter` can signal the `Monitor` by instantiating them and adding a value of 10 to the counter, triggering the monitored milestone: ```shell ❯ curl -s https://durable-app.azurewebsites.net/api/CreateMonitorAndCounter Created Monitor and Counter. Monitor ID: f224fd54-c343-49b6-9305-50e96f6b76d4 Counter ID: e415d7bb-8b6a-4134-a0ba-d81420839657 ❯ curl -s https://durable-app.azurewebsites.net/api/Add\?\ entityId\=e415d7bb-8b6a-4134-a0ba-d81420839657\&amount\=10 Amount added. ``` Logs confirm that the `Monitor` received the signal: ```txt 2024-11-12T12:17:19Z [Information] Progress update: 10.0% ``` The `Counter` and `Monitor` are now communicating dynamically. ## Abstracting Cross-Communication To streamline our implementation, we introduce a helper function for entity signaling. ```kotlin fun TaskOrchestrationContext.signalEntity( entityId: String, operationName: String, operationInput: Any? = null, ) { val operation = Operation(name = operationName, input = operationInput) val signalInput = SignalEntityInput(entityId, operation) this.callActivity("SignalEntity", signalInput) } ``` Using this abstraction, we simplify the `Counter` entity further: ```kotlin if (counter % 10 == 0) { ctx.signalEntity(monitorEntityId, "Update", counter) } ``` This keeps the logic concise and easy to read while ensuring entity signals are handled correctly.. ## What's Next? Now that our entities can communicate with each other, we can move beyond fire-and-forget signals. In the next post, we explore entity calling, allowing entities to send requests and receive responses, further enhancing their capabilities. ## Read Further This post is part of the [Implementing Durable Entities in Kotlin](/implementing-durable-entities-in-kotlin) series. - **Previous**: [Signaling Entities with Input](/implementing-durable-entities-in-kotlin-signaling-entities-with-input). - **Next**: [Calling Entities](/implementing-durable-entities-in-kotlin-calling-entities). --- # Implementing Durable Entities in Kotlin - Calling Entities URL: https://nizar.se/implementing-durable-entities-in-kotlin-calling-entities/ Date: 2025-01-06 So far, our Durable Entities have interacted through signaling, but they lacked direct, two-way communication. In this post, we introduce entity calling, enabling entities and orchestrations to request operations and receive responses. This addition unlocks new possibilities, such as querying state, making informed decisions, and structuring workflows with real-time interactions—all while preserving Durable Functions constraints. ## Introduction So far, we have built a solid foundation for Durable Entities in Kotlin. We have established stateful entities that persist data and allowed them to interact dynamically through signaling. We have also introduced cross-entity communication, where entities can notify each other of significant events. However, all interactions so far have been fire-and-forget. Entities can send signals, but they don't receive immediate responses. This approach is sufficient for many scenarios, but real-world applications often require two-way communication. In this post, we introduce entity calling, which allows entities to invoke operations and receive responses. This unlocks more advanced workflows, such as querying state, making decisions based on results, and chaining interactions in a more structured way. To demonstrate this, we enhance our `Counter` entity to support calls, allowing clients and orchestrations to retrieve its current value. Along the way, we'll introduce best practices for handling entity calls safely with Durable Functions constraints. ## Constraints in Durable Orchestrations In a previous post where we introduced state persistence, we demonstrated how we can query the state of an entity using the custom status metadata. However, as we have previously seen, orchestrations replay their executions to ensure reliability. Any non-deterministic operation, such as fetching a new timestamp or making an external API request, could lead to inconsistent results. Because of this, entity calls need a structured, replay-safe mechanism to avoid breaking the orchestration model. Instead of calling entities directly, we establish a request-response patterns where the caller requests an operation from an entity, the entity processes the request and sends a response, and the caller waits for and retrieves the response. Another challenge is ensuring that responses are mapped to their corresponding requests. Since each entity call runs within an orchestrator that may replay multiple times, using a traditional random ID to track calls would introduce non-determinism. Instead, we need a deterministic way to generate unique event names to safely map responses to their originating requests. ## Building a Request-Response Pattern for Entity Calls ### Extending the Data Model To support entity calls, we extend our data classes. Instead of just sending operation names and input values, we now include metadata for responses. ```kotlin data class Operation( val name: String, val input: Any? = null, val requesterId: String? = null, val responseEventName: String? = null, ) data class SignalEntityInput( val entityId: String, val operation: Operation, val eventName: String? = null, ) ``` The `requesterId` field identifies who made the request, while `responseEventName` specifies the name of the event to which the response should be sent. The `eventName` in the `SignalEntityInput` ensures the result is mapped correctly to its corresponding request. ## Implementing the Call Logic With our extended data model in place, we define a `callEntity` function that facilitates request-response interactions between orchestrators and entities. Unlike fire-and-forget signals, this function ensures that an orchestrator can invoke an operation and reliably retrieve a response. To achieve this, we introduce a structured event-based mechanism that maintains determinism and orchestration safety. When an orchestrator calls an entity, it generates a deterministically unique event name to track its corresponding response. The orchestrator then waits for an external event to receive the result before continuing execution. ```kotlin fun TaskOrchestrationContext.callEntity( entityId: String, operationName: String, operationInput: Any? = null, returnType: Class, ): T { // Generate a deterministically safe and unique response event name val responseEventName = "ResponseEvent_${this.newUUID()}" // Set up the listener for the response event val resultTask = this.waitForExternalEvent(responseEventName, String::class.java) // Generate the payload val operation = Operation( name = operationName, input = operationInput, requesterId = this.instanceId, responseEventName = responseEventName, ) val input = SignalEntityInput(entityId = entityId, operation = operation) // Emit the signal to the entity this.signalEntity(input) // Wait for the response event val resultJson = resultTask.await() // Parse operation input into requested return type and return it val resultOperation = objectMapper.readValue(resultJson) return objectMapper.convertValue(resultOperation.input, returnType) } ``` To maintain determinism, we generate a unique response event name using `newUUID()`, which is safe for orchestration replays as it remains consistent. Using a non-deterministic value like `UUID.randomUUID()` would break replay behavior, making it unsuitable for Durable Functions. Once the event name is generated, we register an event listener using `waitForExternalEvent`. We then package the request into `SignalEntityInput` and send it to the entity. The entity processes the request and emits a response event back to the orchestrator using the stored `responseEventName`. WHen the orchestrator receives the response, it deserializes the result and returns it as the expected data type. ### Enhancing Signal Logic To facilitate structured entity communication, we enhance our signaling logic by introducing an activity function that serves as an intermediary. The `signalEntity` function is now responsible for routing signals safely while ensuring Durable Functions constraints are followed. ```kotlin fun TaskOrchestrationContext.signalEntity(input: SignalEntityInput) { this.callActivity("SignalEntity", input) } @FunctionName("SignalEntity") fun signalEntityActivity( @DurableActivityTrigger(name = "input") input: SignalEntityInput, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ) { ctx.client.signalEntity(input) } fun DurableTaskClient.signalEntity(input: SignalEntityInput) { val operationJSON = objectMapper.writeValueAsString(input.operation) val eventName = input.eventName ?: "EntityOperation" this.raiseEvent(input.entityId, eventName, operationJSON) } ``` By routing signals through an activity function, we ensure that Durable Task Client from within an orchestrator. Additionally, the `eventName` parameter ensures that responses are mapped correctly to their corresponding requests. ### Adding Support for Entity Calls With a safe signaling mechanism in place, we modify our counter to support calls rather than just fire-and-forget signals. Specifically, we introduce a `Get` operation that returns the current counter value to the requester. ```kotlin @FunctionName("Counter") fun counter(@DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext) { var counter = 0 ctx.setCustomStatus(counter) while (true) { val operation = ctx.waitForEntityOperation() when (operation.name) { "Add" -> counter += (operation.input as? Int) ?: 0 "Get" -> { if ( operation.requesterId != null && operation.responseEventName != null ) { val resultOperation = Operation(name = "Result", input = counter) val signalInput = SignalEntityInput( entityId = operation.requesterId, operation = resultOperation, eventName = operation.responseEventName, ) ctx.signalEntity(signalInput) } } "Reset" -> counter = 0 } ctx.setCustomStatus(counter) } } ``` Instead of immediately returning the value, the counter packages the result as an operation and sends it to the requester. The `requesterId` and `responseEventName` ensure that the response is delivered to the correct waiting orchestrator. ### Abstracting the Result Logic We encapsulate the logic for returning results into an extension function: ```kotlin fun TaskOrchestrationContext.returnResult(operation: Operation, value: T) { if (operation.requesterId != null && operation.responseEventName != null) { val resultOperation = Operation(name = "Result", input = value) val signalInput = SignalEntityInput( entityId = operation.requesterId, operation = resultOperation, eventName = operation.responseEventName, ) this.signalEntity(signalInput) } else { throw IllegalArgumentException( "Requester ID or response event name not provided" ) } } ``` With this, we simplify the `Counter` entity further: ```kotlin @FunctionName("Counter") fun counter(@DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext) { var counter = 0 ctx.setCustomStatus(counter) while (true) { val operation = ctx.waitForEntityOperation() when (operation.name) { "Add" -> counter += (operation.input as? Int) ?: 0 "Get" -> ctx.returnResult(operation, counter) "Reset" -> counter = 0 } ctx.setCustomStatus(counter) } } ``` ## Testing Entity Calls ### Creating a CounterCaller Orchestration We define a `CounterCaller` orchestration to test the `callEntity` function: ```kotlin @FunctionName("CounterCaller") fun counterCaller( @DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext, context: ExecutionContext, ) { // Get the ID of the Counter from the input val counterId = ctx.getInput(String::class.java) // For demo: Yield control to ensure the runtime registers everything first ctx.createTimer(Duration.ofSeconds(5)).await() // Call the Get operation on the Counter val counterValue = ctx.callEntity( entityId = counterEntityId, operationName = "Get", returnType = Int::class.java, ) // Log the returned counter value context.logger.info("Received $counterValue") } ``` ### Defining the Orchestration HTTP Trigger We create an HTTP trigger to initialize both the `Counter` and `CounterCaller`: ```kotlin @FunctionName("CreateCounterOrchestration") fun createCounterOrchestration( @HttpTrigger( name = "req", methods = [HttpMethod.GET], authLevel = AuthorizationLevel.ANONYMOUS, ) request: HttpRequestMessage>, @DurableClientInput(name = "ctx") ctx: DurableClientContext, ): HttpResponseMessage { // Create a Counter and a CounterCaller val counterId = Entity(ctx.client, "Counter") val callerId = Entity(ctx.client, "CounterCaller", counterId) // Return an HTTP response containing the IDs val responseBody = """ Created Counter and CounterCaller. Counter ID: $counterId Caller ID: $callerId """ .trimIndent() return request.success(responseBody) } ``` ### Validating the Output We run the setup and check the logs for the retrieved counter value: ```shell ❯ curl -s https://durable-app.azurewebsites.net/api/CreateCounterOrchestration Created Counter and CounterCaller. Counter ID: 3feb953a-0f90-4aaa-81fa-13a96c3c9a6b Caller ID: 7005eee2-b644-4f71-9ec7-b7d18b09bc55 ``` ```txt 2024-11-13T07:17:18Z [Information] CounterCaller: Started 2024-11-13T07:17:18Z [Information] CounterCaller: received 0 ``` ## What's Next? With this we have developed the functionalities that we need to use Durable Entities in Kotlin. In the next and final post in the series, we will refine our implementation by introducing abstractions to simplify interactions and align our approach with the native Durable Entities experience. ## Read Further This post is part of the [Implementing Durable Entities in Kotlin](/implementing-durable-entities-in-kotlin) series. - **Previous**: [Cross-Communication](/implementing-durable-entities-in-kotlin-cross-communication). - **Next**: [Final Enhancements](/implementing-durable-entities-in-kotlin-final-enhancements/). --- # Implementing Durable Entities in Kotlin - Final Enhancements URL: https://nizar.se/implementing-durable-entities-in-kotlin-final-enhancements/ Date: 2025-01-07 Throughout this series, we've explored Durable Entities from the ground up: managing state, creating entities, signaling, cross-communication, and two-way interactions. There steps have culminated in a flexible and functional system for handling stateful workflows in Kotlin. In this final post, we'll refine and enhance our implementation. By introducing abstractions and a DSL-like syntax, we'll improve readability, and aligns it with the native experience. This consolidation will prepare our implementation for real-world usage and future expansions. ## Introduction Throughout this series, we've explored Durable Entities from the ground up: managing state, creating entities, signaling, cross-communication, and two-way interactions. There steps have culminated in a flexible and functional system for handling stateful workflows in Kotlin. In this final post, we'll refine and enhance our implementation. By introducing abstractions and a DSL-like syntax, we'll improve readability, and aligns it with the native experience. This consolidation will prepare our implementation for real-world usage and future expansions. ## Enhancing Durable Entities with Abstractions ### Creating the DurableEntityContext Class We encapsulate common entity operations into a reusable `DurableEntityContext` class. This abstraction simplifies the implementation of entity logic: ```kotlin class DurableEntityContext(val ctx: TaskOrchestrationContext) { var currentOperation: Operation? = null fun waitForOperation(): Operation { currentOperation = ctx.waitForExternalEvent("EntityOperation", String::class.java) .await() .let { objectMapper.readValue(it) } return currentOperation ?: throw IllegalStateException("Operation could not been initialized.") } inline fun getState(): T { return ctx.getEntityState(ctx.instanceId, T::class.java).await() ?: throw IllegalStateException("Entity state has not been initialized.") } fun setState(newState: Any?) { ctx.setCustomStatus(newState) } inline fun getInput(): T { return T::class.cast(currentOperation?.input) } fun returnResult(value: T) { val operation = currentOperation ?: throw IllegalStateException( "No operation in progress to return result for" ) val (requesterId, responseEventName) = operation.requesterId to operation.responseEventName if (requesterId != null && responseEventName != null) { val resultOperation = Operation(name = "Result", input = value) val signalInput = SignalEntityInput( entityId = requesterId, operation = resultOperation, eventName = responseEventName, ) ctx.signalEntity(signalInput) } else { throw IllegalArgumentException( "Requester ID or response event name not provided" ) } } } ``` ## Retrieving Entity State with Activities ### Creating the getEntityState Function We define a helper function for retrieving the state of an entity using an activity: ```kotlin fun TaskOrchestrationContext.getEntityState( entityId: String, returnType: Class, ): Task { return this.callActivity( "GetEntityState", GetEntityStateInput(entityId), returnType, ) } @FunctionName("GetEntityState") fun getEntityStateActivity( @DurableActivityTrigger(name = "input") input: GetEntityStateInput, @DurableClientInput(name = "durableContext") durableContext: DurableClientContext, ): Int? { return durableContext.client .getInstanceMetadata(input.entityId, true) ?.readCustomStatusAs(Int::class.java) } ``` ## Refactoring the Counter Entity ### Updating the Counter Logic With the `DurableEntityContext` abstraction, the `Counter` entity is simplified: ```kotlin @FunctionName("Counter") fun counter( @DurableOrchestrationTrigger(name = "taskCtx") taskCtx: TaskOrchestrationContext ) { val ctx = DurableEntityContext(taskCtx) ctx.setState(0) while (true) { ctx.waitForOperation() when (ctx.currentOperation?.name?.lowercase()) { "add" -> ctx.setState(ctx.getState() + ctx.getInput()) "reset" -> ctx.setState(0) "get" -> ctx.returnResult(ctx.getState()) } } } ``` ## Adding State Rehydration ### Initializing State To ensure the entity state is properly initialized, we define an `initializeState` function: ```kotlin fun initializeState(initialValue: Any?) { val state = ctx.getEntityState(ctx.instanceId, Any::class.java).await() if (state == null) { ctx.setCustomStatus(initialValue) } } ``` ## Introducing a DSL-like Syntax ### Simplifying the Entity Loop We encapsulate the entity loop in the `DurableEntityContext`: ```kotlin fun runEntityLoop(handleOperation: DurableEntityContext.() -> Unit) { while (true) { waitForOperation() this.handleOperation() } } ``` ### Abstracting the Entity Function We introduce an `asEntity` function to further streamline entity definitions: ```kotlin fun asEntity( taskCtx: TaskOrchestrationContext, initialState: Any? = null, block: DurableEntityContext.() -> Unit, ) { val ctx = DurableEntityContext(taskCtx) ctx.initializeState(initialState) ctx.runEntityLoop { block(ctx) } } ``` ### Final Entity Implementation The Counter entity now has a clean and expressive syntax: ```kotlin @FunctionName("Counter") fun counter(@DurableOrchestrationTrigger(name = "ctx") ctx: TaskOrchestrationContext) { asEntity(ctx, initialState = 0) { when (currentOperation?.name?.lowercase()) { "add" -> setState(getState() + getInput()) "reset" -> setState(0) "get" -> returnResult(getState()) } } } ``` ## Conclusion With this final post, we have refined our Durable Entities implementation, making it cleaner, more expressive, and better suited for real-world usage. By introducing abstractions like DurableEntityContext, we reduced boilerplate and improved maintainability. We also added state rehydration and entity state retrieval via activities, ensuring consistency and reliability. Finally, a DSL-like syntax makes entity definitions more intuitive and streamlined. This concludes our series, having covered all fundamental functionalities. The implementation provides a solid foundation and can be further optimized for performance. ## Read More This post is part of the [Implementing Durable Entities in Kotlin](/implementing-durable-entities-in-kotlin) series. - **Previous**: [Calling Entities](/implementing-durable-entities-in-kotlin-calling-entities). --- # How I Use LLMs as a Developer URL: https://nizar.se/how-i-use-llms-as-a-developer/ Date: 2025-04-25 I have been using Large Language Models (LLMs) for the last two years. During this time, I’ve noticed that people’s opinions about LLMs vary dramatically. The community seems split—some developers swear by them, while others remain skeptical or outright dismissive. Personally, I find these differences fascinating because they highlight that there’s no single “right” way to think about LLMs, and definitely not a single right way of using them. Instead, the value you get largely depends on how you choose to approach them. I have been using Large Language Models (LLMs) for the last two years. During this time, I’ve noticed that people’s opinions about LLMs vary dramatically. The community seems split—some developers swear by them, while others remain skeptical or outright dismissive. Personally, I find these differences fascinating because they highlight that there’s no single “right” way to think about LLMs, and definitely not a single right way of using them. Instead, the value you get largely depends on how you choose to approach them. Over the past year, I’ve had several conversations where people asked me how I actually use LLMs in my day-to-day work. Rather than repeating myself, I decided to write it down. This post is a reflection of my own journey–how my usage has evolved, what I’ve learned, and how LLMs have changed the way I work as a developer. ## Exploring and Experimenting When I first started out, I was mainly curious about what these models could actually do. I tried asking them to generate tests, code snippets, and documentation. It was impressive, but I quickly encountered familiar pitfalls: hallucinations, [prompt drift](https://www.marilynfrank.com/word/prompt-drift), and context or output length limitations. The rapid pace of model updates also introduced noticeable shifts–some users even complained that the models felt “lazy” or less helpful after certain updates. Interestingly, I noticed that I consistently got better results when saying “please” and “thank you”. At first, this seemed silly, but later studies confirmed that polite interactions genuinely influence result quality. Eventually, I realized it wasn’t merely about emotional pleading–being deliberate and precise in my communication made the real difference. Clearly stating goals, providing explicit context, and sharing relevant examples consistently led to improved outcomes. Soon, my interactions became naturally conversational. I’d clearly describe an issue, provide tests or code snippets, and outline exactly what I am aiming to achieve. If the model made incorrect assumptions, I’d nudge it gently in the right direction. If its suggestions were suboptimal, I’d ask clarifying questions. Over time, this approach felt less like prompting and more like a technical discussion. ## Dealing with Context Drift As conversations lengthened, context drift became noticeable. The model would occasionally lose track of details provided earlier. However, as someone who practices Test-Driven Development (TDD), this wasn’t a major issue. Breaking problems into smaller increments naturally prevented context overload. Between iterations, I’d briefly remind it of critical details or qualities I aimed for. While structured prompting greatly improved the results, it also made my prompts longer and more cumbersome over time. ## Becoming A Better Technical Communicator With practice, I learned to ask clearer, more concise questions without needing lengthy examples or constant repetition of context. It pushed me to become more precise when articulating technical challenges, patterns, and specifications–ultimately sharpening my technical communication overall. For example, instead of pasting a block of code and asking: “I have this code that maps over observables and applies some logic, but it doesn’t seem to emit anything in some cases. Any idea what might be wrong?” I might now ask something like: “I’m working on a data transformation pipeline that filters and maps over nested observables. Under certain conditions, the inner stream completes before emitting. What are some clean strategies to avoid silent drops in that case?” Or if I’m exploring alternatives to a design: “How could we generalize this pattern so it supports multiple input types while keeping the transformation logic decoupled from the source format?” Being able to formulate questions like these, without dragging in unrelated implementation details, has helped me communicate more effectively overall. ## Why I Still Write My Own Code An important principle I’ve always adhered to is never copy-pasting code written by others into my projects. Naturally, this extended to code generated by LLMs. Many developers happily embrace direct code generation, and perhaps someday I’ll reconsider, but right now, it just doesn’t feel right to me. Maybe it’s philosophical, but I firmly believe the code I introduce should be code that I have personally written. This principle has shaped my relationships with LLMs significantly. Rather than using them for code completion or direct generation, I treat them as conversational partners–tools for discovery, prototyping, rubber-ducking, and exploring curiosities. My interactions typically begin when I sense potential improvements or approaches I haven’t yet considered. I ask questions, explore possibilities, and deepen my understanding. Only once I clearly see the solution, and fully grasp it myself, do I write the actual code. ## Do LLMs make us Smarter or Lazier? This approach brings up the classic question: Does using calculators, search engines, or LLMs make us smarter or lazier? Personally, I think it depends entirely on the mindset. Imagine working alongside smarter, more experienced colleagues–do you delegate the difficult tasks to them, or do you actively learn from their expertise? That’s how I view my relationship with LLMs: an opportunity to question, learn, and become a better developer. ## What’s Next on My Journey? The pace of innovation around LLMs continues to astonish me. I’m particularly eager to explore several emerging tools and practices: - Multi-Document Context \ Allows models to pull in relevant information from multiple sources, helping them reason more effectively across large or fragmented codebases and documents. Instead of giving all the context upfront, you define reusable instructions or rules, and the model knows where to look and what to apply. ([awesome-mdc](https://github.com/benallfree/awesome-mdc/blob/main/what-is-mdc.md)) - Task Management System \ These help break complex projects into smaller, manageable tasks and orchestrate their execution effectively. ([Claude Task Master](https://github.com/eyaltoledano/claude-task-master), [Boomerang Tasks](https://docs.roocode.com/features/boomerang-tasks)) - Model Context Protocol \ A protocol that lets models seamlessly integrate with various tools and data sources, dramatically expanding their capabilities. ([MCP Introduction](https://modelcontextprotocol.io/introduction)) - Browser Automation for AI \ Makes websites directly accessible to AI agents, greatly expanding how models can interact with web applications. ([Browser Use](https://docs.browser-use.com/introduction)) - AI Integration with GitHub Actions \ Integrating AI models directly into development workflows, enriching and automating parts of the CI/CD process. ([GitHub Actions](https://docs.github.com/en/github-models/integrating-ai-models-into-your-development-workflow#using-ai-models-with-github-actions)) Admittedly, adopting some of these new technologies might challenge my current principle of writing all the code myself. Nevertheless, I’m eagerly looking forward to seeing where this journey takes me next. --- # Agentic TDD URL: https://nizar.se/agentic-tdd/ Date: 2025-06-17 Since I started programming, I've insisted on always writing my own code. It wasn't out of necessity; it was a personal choice. Even as LLMs (Large Language Models) became part of my workflow, I stayed firm about handling production code myself. Last week, that changed. Since I started programming, I've insisted on always writing my own code. It wasn't out of necessity; it was a personal choice. Even as LLMs (Large Language Models) became part of my workflow, I stayed firm about handling production code myself. Last week, that changed. ## Exploring Coding Agents After seeing countless demos online and reading about new workflows, I decided it was time to experience an AI-powered coding agent myself. I subscribed to Claude Code and jumped into a FigJam-inspired whiteboard project. To my surprise, Claude Code handled almost everything: it suggested dependencies, initialized the project, set up tools, configured GitHub CI and security workflows, and even proposed a detailed project plan with tasks. It's one thing to watch demos; it's another to be on the receiving end of an agent that understands your intentions. Experiencing it firsthand was genuinely surprising. There was something uniquely special about how it interacted and adapted to my needs. ## First Encounter with Agentic TDD That sense of novelty really hit home when I guided Claude Code through Test-Driven Development (TDD). At first, I needed to spell out the process clearly in its instructions. But once the guidelines were set, watching it autonomously follow TDD principles was unexpectedly profound. It reminded me of hearing the [first song sung by a computer](https://en.wikipedia.org/wiki/Daisy_Bell): unexpected, imperfect, but a glimpse of something bigger. ![Demo of Claude Code doing TDD](/uploads/claude-code-tdd.gif) To help Claude Code stay on track, I provided it with a pre-commit script that handled formatting, linting, and running both unit and integration tests. This made the agent's workflow more consistent and reduced the need for explicit guidance. ## Where the Agent Falls Short The initial excitement, however, was quickly tempered by reality. It didn't take long to see why software engineers remain essential. I was still responsible for critical architectural decisions and testing strategies, which is especially tricky when working with canvas-rendered elements that can't be easily tested with standard DOM queries. I often found myself balancing simplicity and complexity. Sometimes, Claude Code defaulted to "safe" behaviors: adding new functions instead of refactoring existing ones (likely to avoid breaking working code), or skipping refactoring in test code unless prompted. Personally, I prefer tests to remain clean and concise—a discipline I picked up from senior colleagues at [factor10](https://factor10.com/). Extracting common setup logic into helpers keeps tests readable and maintainable. Other times, the agent's shortcuts forced me to intervene. For example, Claude Code disabled a linting rule instead of correcting a typing issue. It also tried to revert a package upgrade because the new version required configuration changes. In both cases, I had to step in, clarify my preferences, and update its guidelines. ## Putting it to Work in Production Wanting to see how it performed in the real world, I started using Claude Code in a client project where we're evaluating AI tools for developer productivity. In these larger, practical scenarios, it performed better than I expected. With some deliberate guidance, it proved invaluable for rapid prototyping and managing huge amounts of information. I still found myself "babysitting" a bit: refining test helpers, nudging it towards architectural simplifications, and occasionally pulling it back on track. Still, the early results were genuinely encouraging, both for the client and the team. What stood out wasn't just the speed from idea to prototype, but how the agent made discovery and iteration less tedious. Tracing logic across dozens of files, validating assumptions, and testing multiple hypotheses all became much faster. While the agent's code wasn't always production-ready, having something concrete to refine often surfaced improvements and insights that only became apparent once I could interact with a real implementation. ## Productivity Gains & Their Costs This productivity boost raised some interesting tradeoffs. I could spend the saved time on quality—optimizing performance, refining tooling, or enhancing user experience—while keeping up the same pace. Or, I could use the boost to simply get more done, faster. Right now, I'm balancing both increased speed and quality. It's not that the agent produces better code than I could have, or solves problems I couldn't. The real value is how quickly I can move through possibilities, validate approaches, and focus my attention where it matters. Sometimes it suggests alternatives I hadn't considered, but the biggest gain is simply more time and space to refine and improve ideas. But as more work got done in less time, bottlenecks shifted elsewhere. Code reviews, in particular, became a new pinch point. Personally, I'm not a big fan of traditional pull request workflows; the important design decisions should ideally happen upfront through pair programming or quick collaborative discussions. Real-time collaborations clarify intentions and issues before they snowball. Test and pipeline performance can quickly become a bottleneck. With TDD, tests are run multiple times for every change, and this overhead grows even more noticeable when working agentically. If your test suites or pipelines aren't well-optimized, you'll quickly feel the drag. ## Why Agentic TDD Maybe agentic TDD isn't for everyone, and that's fine. Personally, TDD keeps me productive, effective, and sane. It might slow down the agent, but it also ensures I understand, agree with, and can trace the changes being made. The reasoning and the tests the agent creates give me context I need to follow along or step in when something feels off. I also wonder if TDD could reduce the risk of introducing "protected" or memorized code, though I can't say for certain. Either way, it's an intriguing thought. ## Conclusion Looking back, I'm struck by how quickly agentic coding became mainstream. It happened almost overnight. Not long ago, making the most of LLMs meant we had to manage the plan ourselves: track tasks in our heads, handling each step manually, and guiding the model by reviewing outputs and feeding back the results. We were the bridge between the model and the tools, stitching everything together one prompt at a time. Now, that same [loop runs inside the agent](https://philz.dev/blog/agent-loop/). Instead of holding the task list in mind and steering each operation, we hand over the process. It's remarkable how such a small shift in workflow—passing off this mental orchestration—has had such an outsized impact. Similarly, the [MCP (Model Context Protocol) loop](https://www.anthropic.com/news/agent-capabilities-api) is both simple yet impactful. Claude Code retrieves available tools, reasons about appropriate actions, executes tool calls, and iterates until successful. There are still times I want to just make changes myself, and I'm figuring out ways to minimize those interruptions. Of course, some limitations are tough to work around. For example, when Claude Code tries to rename things across files with `grep` or `regex` and stumbles on syntax variations. Even so, Claude Code hasn't just made me fast—it's changed how I think about building software. The real shift is realizing that good results and confidence in my work don't depend on typing every line myself. I still review and shape the code, but I can now focus my effort where it matters the most: iterating more quickly, exploring new options, and making higher-impact decisions. I'm still learning to collaborate with it, and I'm curious to see where this journey goes next. But I already know one thing: I don't want to go back. ## Practical Tips If you're curious about Claude Code or are already experimenting, here are personal tips: - Interrupt early if the agent veers off track. - Commit frequently. You can also ask it to undo changes. - Compact context between large tasks for best results. - Regularly refine instructions, something I admittedly don't always prioritize enough. Things to try: - Use [MCP servers](https://github.com/punkpeye/awesome-mcp-servers) to provide the agent with direct feedback. - Try the planning mode for investigative tasks. - Share screenshots to clarify complex tasks. - Leverage tools like [Prompt Improver](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prompt-improver). - For advanced users: experiment with the [git worktree pattern](https://github.com/anthropics/claude-code/issues/1052). If you try agentic TDD, be sure to give the agent clear, explicit instructions to prevent shortcuts. Otherwise, it might begin implementing as soon as any test fails, regardless of the reason, or pack too many assertions into a single test. I found the following instructions helpful to include: - Always start with a test that fails for the correct reason. - Write only one test at a time. - Use a single assertion per test. - Verify test infrastructure before testing actual behavior. - Provide stub implementations to avoid errors unrelated to actual behavior. - Tests should fail due to incorrect behavior, not on missing methods/properties. - Implement the minimal necessary code to pass the test. - Don't add properties, methods, or logic that aren't required by the current failing test. - Once the test passes, refactor both test and implementation code. Lastly, check out the official [Claude Code Best Practices](https://www.anthropic.com/engineering/claude-code-best-practices) guide. It's genuinely helpful. --- # TDD Guard for Claude Code URL: https://nizar.se/tdd-guard-for-claude-code/ Date: 2025-07-21 Claude Code fit well into my workflow, but sticking to Test-Driven Development (TDD) principles still needed occasional reminders: one test at a time, fail first, no over-implementation. It worked, but it wasn’t effortless. So I built TDD Guard, a utility that handles that for me and keeps my attention focused on designing solutions rather than policing TDD. Claude Code fit well into my workflow, but sticking to Test-Driven Development (TDD) principles still needed occasional reminders: one test at a time, fail first, no over-implementation. It worked, but it wasn’t effortless. So I built [TDD Guard](https://github.com/nizos/tdd-guard), a utility that handles that for me and keeps my attention focused on designing solutions rather than policing TDD. ## Leveraging Hooks The breakthrough occurred when Anthropic announced [Hooks](https://docs.anthropic.com/en/docs/claude-code/hooks) for Claude Code—coincidentally, the very day I was seeking a better solution. Hooks automatically execute commands based on predefined conditions, such as before or after an agent’s action, making them ideal for enforcing coding standards. I began by creating a hook that intercepts all file modification operations before execution, triggering validation checks for common TDD violations: - Implementing functionality without a relevant failing test - Implementing more than necessary to pass a test - Adding more than one test at a time The validator integrates hook event data, the agent’s current todo list, and the latest test run output. It then invokes a separate Claude Code session to verify adherence to TDD principles. Since Claude Code hooks run as separate processes, TDD Guard persists context data to files between each phase. This approach allows different hooks, like TDD validation before a change and lint checks after, to access shared state without relying on complex inter-process communication. It keeps the system reliable and easy to reason about. If no violations are detected, the hook proceeds without interference. But when a violation is found, it blocks the action and returns feedback clearly stating the issues, along with corrective guidance. ## Dogfooding the Development With the prototype integrated directly into my workflow, the improvements were immediate. Eliminating manual reminders was a refreshing change. But the agent's behavior initially felt too compliant. I began to suspect this wasn't due to the validation logic itself, but rather because the agent had absorbed all the TDD-related terminology present throughout the project—things like test names, helper functions, and prompt documents. It seemed to follow TDD principles simply because the surrounding context steered it that way, not because enforcement was actually working. To properly assess the enforcement, I needed to see how the system behaved outside a TDD-saturated environment. Testing it in an unrelated project confirmed my suspicion. Some clear violations slipped through, while valid actions were mistakenly blocked, sometimes prompting humorous attempts by the agent to circumvent restrictions using terminal commands. ## Integration Testing That prompted me to invest in robust integration tests. I implemented comprehensive test data factories covering various scenarios: - Empty, in-progress, completed, or irrelevant todo lists - Diverse test outputs, including empty, irrelevant passes, and various failure types - Modifications including minimal implementations, excessive functionality, multiple tests, and diverse refactoring cases This strategy enabled easy adjustments to data structures and facilitated quick support for new languages like Python without extensive test rewrites. These tests also accommodated Claude Code’s distinct modification tools: Write, Edit, and MultiEdit. ## Context Engineering Each of Claude Code's file operation tools behaves differently. Write generates complete file contents, while Edit and MultiEdit apply targeted modifications to existing code. These differences made static prompts feel inefficient and misaligned across many cases. One-size-fits-all prompts often led to confusing or conflicting behavior, especially when only some context was relevant to the operation. For example, Edit or MultiEdit might attempt to introduce three tests, two of which already existed and were simply refactored. In such cases, the prompt needed to include guidance for identifying genuinely new tests. But the same guidance would be irrelevant for a Write operation, which always starts from a clean state. To address this, I built a dynamic system that assembles only the relevant instruction modules based on the operation type and situation. This ensures the model receives clear, concise context tailored to the task at hand. ## Multi-Model Support As test complexity increased, slow feedback loops began to limit how quickly I could iterate. To speed things up, I added a new validation client that runs prompts directly against Anthropic's API, bypassing the Claude Code CLI. This made integration tests faster and more practical in CI workflows. It also gave me the flexibility to experiment with different models to balance cost, speed, and performance. This setup also exposed a few practical challenges. Some models included verbose explanations or inline code before finalizing an answer, which tripped up the parser and caused false positives. TDD was a big help here too. I could reliably reproduce the bug before making any fixes, which made it easy to confirm that the parser now handles that behavior correctly. ## Rules vs. Mindset Despite improvements, scenario tests still exhibited inconsistencies due to subjective model interpretations of TDD principles, notably during refactoring. While explicit rule enforcement resolved test inconsistencies, it only achieved mechanical adherence that did not translate into improved software quality. I discovered that without explicit guidance, agents naturally skip the refactoring phase of the red-green-refactor cycle or at best make only superficial changes. To understand this better, I had Claude Code implement a shopping cart using only TDD Guard's blocking mechanism, without any instructions about software quality or testing practices. The results were telling: while the guard successfully enforced test-first development, the resultant code still suffered from tight coupling, duplication, and poor overall design. This drove home that TDD's value comes from the mindset and discipline it instills, not from mechanical rule-following. Inspired by my mentor, Per at factor10, who recently shared [reflections on this topic](https://programmaticallyspeaking.com/a-tdd-mindset.html), I began exploring how to encourage a genuine TDD mindset. Could aspects of this mindset be emulated using hooks? ## Meaningful Refactoring Initially, I considered employing a reviewing model to suggest meaningful refactorings. However, lacking sufficient system-wide context, this approach risked inadvertently increasing complexity rather than simplifying the solution, a common problem with local rather than global optimization. Providing comprehensive context to the model proved both impractical and prohibitively expensive. Instead, I integrated lightweight linting tools such as Sonar to identify complexities and code standard violations, prompting refactoring tasks. Unfortunately, post-action hooks were weakly enforced, frequently resulting in deferred or ignored refactoring tasks. To strengthen enforcement, I stored identified issues, mandating resolution in subsequent pre-action validation phases. Deliberately withholding specific issue details initially encouraged the agent to attempt meaningful refactorings independently. Persistent issues were later explicitly communicated as feedback. ## Balancing Art and Science Defining "meaningful refactoring" remains inherently subjective and context-dependent, with varying interpretations of quality standards, design principles, and complexity among developers and teams. Finding practical tools to flag unnecessary complexity proved trickier than expected. Most are either correlated with line count or require extensive setup that makes them impractical. Moreover, automated linting rules primarily encouraged superficial changes, such as smaller function sizes, rather than coherent functionality. Recognizing this limitation, I concluded TDD Guard should enforce basic linting rules during refactoring cycles while leaving deeper, meaningful design considerations to the developers themselves. While the hook improves results, the system-thinking and design awareness that human developers bring cannot be matched. After all, engaging actively in system design and architecture remains one of programming's most enjoyable aspects. ## Wrapping up The TDD Guard project began two weeks ago during a relaxed, collaborative yearly event, a BBQ at [Jimmy](https://jimmynilsson.com/)’s place with colleagues from [factor10](https://www.factor10.com/). The initial prototype, a simple “cat detector” blocking modifications containing “mew”, served as a quick validation of core concepts. I intentionally retained these early explorations in the project’s history as a testament to something I deeply believe in: that joy, authenticity, meaningful companionship, and psychological safety nurture random curiosity and creativity into tangible innovations. Applying TDD principles and dogfooding throughout the development enabled rapid iterations and valuable community contributions, including Python support. The community’s warm reception, enthusiastic feedback, and collaborative spirit have been profoundly motivating. I'm especially grateful to my mentor [Martin](https://recurse.se/), whose encouragement and belief in the idea helped bring it to life. Feel free to explore [TDD Guard](https://github.com/nizos/tdd-guard), and please reach out with feedback or contributions. Your insights are invaluable and greatly appreciated. --- # Lean Claude Code for Production URL: https://nizar.se/lean-claude-code-for-production/ Date: 2025-08-15 Many agentic coding demos showcase landing pages, one-off scripts, or greenfield projects. While technically impressive, they often produce single files with little thought for maintainability. Even OpenAI's ChatGPT 5 launch demo followed this pattern. At the other extreme, some content focuses almost entirely on elaborate integrations and orchestrations. Many agentic coding demos showcase landing pages, one-off scripts, or greenfield projects. While technically impressive, they often produce single files with little thought for maintainability. Even OpenAI's ChatGPT 5 launch demo followed this pattern. At the other extreme, some content focuses almost entirely on elaborate integrations and orchestrations. Both have their place, but neither reflect how I ship production code with agents. That gap in middle-ground coverage feeds two common misconceptions: - AI-generated code cannot meet disciplined, production-grade standards. - Agentic coding is inherently complex and needs heavy upfront investment. Here's the middle path I use in real projects. Simple setup, clear guardrails, deliberate steering, and no ceremony. ## Project Setup Claude Code keeps project context and details in `CLAUDE.md`, usually created with the `/init` command when you first start working on a project. A common trap I've fallen into myself is stuffing it with every rule and best practice. It feels logical. You want the agent to "know everything" upfront, but in practice, it often backfires. When the file contains many goals and constraints at the same level, the model struggles to prioritize. This isn't forgetfulness—it's how large language models work when instructions compete for attention. This increases inconsistency and makes behavior feel frustratingly unreliable. What works for me is keeping `CLAUDE.md` lean, ideally under 100 lines, focusing on: - The project's context, purpose, and tech stack - A brief project structure - Common scripts to run Here's roughly what I include: ```md # Project Name A brief description of what this project does. ## Structure - src/validators/ # Input validation patterns - src/middlewares/ # Express middleware - src/utils/ # Shared utilities and helpers - test/factories/ # Test data factories ## Commands npm run test # Run all tests npm run test:unit # Run unit tests only npm run lint # Check code quality ``` This lean file helps the agent with things it typically struggles with or would waste time and tokens figuring out repeatedly: - Understanding the domain and aligning execution with the project's purpose - Avoiding costly full-codebase scans to find existing interfaces, implementations, and helpers - Knowing which scripts to run without having to look them up each time This isn't about lowering your coding standards: abandoning design patterns, functional programming principles, or testing requirements. It's about enforcing these standards through guardrails rather than bloated instructions. ## Devcontainers Even in a simple setup, [devcontainers](https://docs.anthropic.com/en/docs/claude-code/devcontainer) are worth it. They provide a consistent, reproducible, and isolated environment with all project dependencies pre-installed and pre-configured. They prevent clashes between system and project dependencies and can gate network access with a whitelist firewall. ## Code Quality ### Test-Driven Development The biggest quality gain comes from enforcing strict Test-Driven Development (TDD): 1. Write a failing test first. 2. Write the minimal code to pass. 3. Refactor while keeping tests green. Agents are good at producing code quickly. They are less consistent at sequencing work. TDD supplies the sequence. It also gives short, objective feedback loops that the agent can reason about. Without this structure, agents tend to: - Jump straight to implementation code - Write all tests at once - Skip test runs - Skip the refactoring step - Implement more than the current test requires The result is untested code, poor design decisions, over-engineering, and technical debt. To make TDD non-negotiable, I built [TDD-Guard](https://github.com/nizos/tdd-guard) for Claude Code. It blocks violations such as implementing without a failing test, adding multiple tests at once, or over-implementing. Beyond TDD, it can enforce linting rules, like strict functional programming patterns, ensuring the agent follows your coding standards automatically. With these rules enforced automatically, I focus on the problem and the design instead of policing process. TDD itself encourages better design, thorough test coverage, and creates living documentation of your codebase. ### Linting, Formatting, and Commits Similarly, I use [husky](https://github.com/typicode/husky) with [lint-staged](https://github.com/lint-staged/lint-staged) to automatically lint and format code before commits. With [commitlint](https://github.com/conventional-changelog/commitlint), I enforce conventional commit messages. This frees up more context and produces more consistent behavior. ## Planning Work Claude Code's planning mode is where I usually start. Here, the agent focuses on gathering information and mapping out the steps to complete a task. The key is to be deliberate and explicit in your instructions, while pointing it to the files and references it needs. Instead of verbose guidelines on testing and design, I show it an existing, well-structured component and its tests as an example. When the plan is ready, I don't obsess over perfection. The real value is in the context the agent has gathered. I only intervene at this stage if it clearly missed an important aspect or if I spot signs of over-engineering. It's usually more effective to address issues as they arise rather than front-loading all corrections. Finally, I rarely use auto-accept mode. Even though it can be faster, manually approving or steering each step gives me more control and helps catch small deviations before they snowball. ## Managing Context Models have a limited context window—think of it as their working memory. As it fills up, performance gradually degrades. Planning mode is excellent for gathering context without overloading the main agent with irrelevant details. It can delegate investigation to subagents, which read the relevant files, filter out noise, and pass back only the important findings. This keeps the main agent's context lean and focused. You'll see Claude Code using this strategy when it performs a `Task`. You can use the same strategy in any mode by simply asking it to use a subagent to do any investigation work you need. Even with this strategy, you'll eventually reach the limit. When that happens, Claude Code will offer to compact the context, replacing it with a summary. The problem is that summaries aren't perfect. They sometimes prioritize the wrong things or leave out critical details. This is especially true when priorities shift mid-session. For example, you might spend most of the session on trial-and-error, then pivot after a breakthrough. If the compacted summary still focuses on the earlier, discarded work, the agent's future decisions will be skewed. To avoid this, I give custom instructions when triggering `/compact`, telling the agent exactly which details to preserve. This keeps the session better aligned with the findings I value. Sometimes my work spans multiple sessions, or I've built up a context I want to preserve for later. In these cases, I ask the agent to save the current context or findings to a temporary markdown file, essentially a session-specific `CLAUDE.md`. Before we finish, I often have the agent propose updates to this document based on our progress or new findings. This quick review step removes noise, clarifies ambiguous points, and ensures the saved version captures only what's worth carrying forward. In the next session, I point the agent to that file and have it read it in full, followed by gathering any additional information it needs. This approach avoids relying on compaction summaries and gives a clean, intentional handoff between sessions. ## Bottom Line I've been using this setup for the last couple of months, and I'm very happy with the results. Custom agents, personas, MCPs, and orchestration can help in specific cases. You most likely don't need them to start, especially if your main objective is business value and software quality. Prompt optimization has limited return here, especially when you already know the outcome you want. The bigger gains come from practice and building intuition for effectively interacting with agents. For me, the formula is simple: - Be deliberate and explicit about what you want to achieve. - Give the agent an efficient way to find the information it needs. - Point it towards relevant examples of the approach you want it to follow. If work drifts, stop it early, explain the correction, and continue. Claude Code also lets you press `Escape` twice to roll back to a previous point in the session. No elaborate orchestration needed—just lean, deliberate practice. --- # The Code is the Context URL: https://nizar.se/the-code-is-the-context/ Date: 2025-12-01 Most advice on working with agents focuses on orchestration and interaction strategies. But I keep finding renewed significance in the software development fundamentals we've always known. After all, agents don't just respond to how we prompt them, they also respond to what they find in the codebase. Most advice on working with agents focuses on orchestration and interaction strategies. But I keep finding renewed significance in the software development fundamentals we've always known. After all, agents don't just respond to how we prompt them, they also respond to what they find in the codebase. ## Discovery Agents start each session fresh, with no memory of previous work, just the prompt and whatever context files we provide. So they search, read, and infer, assembling context from what they find. This discovery process is constant. When writing a test, they search for patterns to follow. When a fix fails, they explore to understand why. When preparing a commit message, they read changes to summarize intent. You can shape discovery through how you interact with and orchestrate the agents: - **Delegation** keeps the main context clean. Subagents handle exploration and return only what's relevant. - **Progressive disclosure** provides information just-in-time rather than front-loading everything. - **Feedback loops** catch mistakes early. The faster agents can verify their work through tests, type checks, and linting, the less they diverge. I cover these in my [TDD Guard webinar](https://factor10.com/news/on-demand-webinar-5-pro-tips-from-developing-and-using-tdd-guard/). But interaction strategies have limits. The deeper factor is the codebase itself: unclear code forces more searching, disorganized systems lead to false assumptions, and complexity requires lengthy explanations. ## What's Old Becomes New Again When you work closely with a codebase for months or years, you develop home blindness. The convoluted function that "works fine". The abstraction that made sense at the time. The naming obvious to you but obscure to anyone else. Agents don't have your context. They see the code as it is, not as you remember it. Watch where they search repeatedly for the same concept. Notice when they misunderstand what a function does despite reading it. Pay attention when they need long explanations to understand concepts and constraints the code should express. That friction is information. It points to familiar territory: - **Self-documenting code** means less searching. When the functions and variables reveal their intent, agents don't need to trace through their implementations. - **Single responsibility** means fewer surprises. When a function does one thing, assumptions hold. Fewer surprises means less recovery and backtracking. - **Simplicity** means faster reasoning. Complex solutions require more context to understand. The less there is to infer, the faster discovery completes. - **Tests** serve as executable documentation. Well-written tests tell agents what the code should do without requiring archaeology. These aren't new. Agents make them visible in a new way. ## Closing Thoughts This isn't a call to refactor your entire codebase. It's an invitation to notice what the agents reveal, then act on it where it matters. The changes that help agents discover faster also make code easier to review, reason about, and maintain. Thanks to my colleagues at [factor10](https://factor10.com/) for the conversations that shaped these ideas, especially [Jimmy Nilsson](https://jimmynilsson.com/), whose insight that "friction is information" became central to this piece.