Bruno Calza - Blog

Building a Grow-Only Counter on a Sequentially Consistent KV Store

2026-04-13T15:15:07+00:00

This post is part of a series on Fly.io’s distributed systems challenges:

Implementing Snowflake Unique ID Generation

Generating Unique IDs with Raft Consensus

Fly.io’s Broadcast Challenges

Building a Grow-Only Counter on a Sequentially Consistent KV Store (this post)

We’re going to discuss Challenge #4: Grow-Only Counter. This challenge is particularly tricky. I wouldn’t say it’s hard, but if the goal is to learn, there’s a lot to unwrap.

The task is to build a grow-only counter. Nothing strange so far. However, the specification says to build the counter on top of Maelstrom’s SeqKV built-in service. And that’s where things get weird. In this post I try to explore that weirdness to the best of my ability. Moreover, I briefly touch on CRDTs (Conflict-free replicated data type).

First, let’s understand what is going on. The following test needs to pass:

./maelstrom test -w g-counter \
    --bin ~/go/bin/maelstrom-counter \
    --node-count 3 \
    --rate 100 \
    --time-limit 20 \
    --nemesis partition

The g-counter workload sends two types of request: add and read, that our nodes will need to accept.

# add request
{
  "type": "add",
  "delta": 123
}
# add response
{
  "type": "add_ok"
}

# read request
{
  "type": "read"
}
# read response
{
  "type": "read_ok",
  "value": 1234
}

Essentially, the test checks that after all adds are done, every node’s final read sees the full sum.

SeqKV is a key-value store that the node can use to build the algorithm. SeqKV offers the following API:

ReadInt(key string) -> int
Write(key string, value any)
CompareAndSwap(key string, from any, to any, createIfNotExists bool)

And that’s all we need to get started.

Starting simple

Let’s use the key counter in our key-value store to store the value of our counter. So we initialize our nodes with counter set to 0.

n.Handle("init", func(msg maelstrom.Message) error {
    return kv.Write(context.Background(), "counter", 0)
})

Then whenever an add request comes in, we read the value, update it, and write it back:

n.Handle("add", func(msg maelstrom.Message) error {
    var req struct {
        Type  string `json:"type"`
        Delta int    `json:"delta"`
    }
    if err := json.Unmarshal(msg.Body, &req); err != nil {
        return err
    }

    old, err := kv.ReadInt(context.Background(), "counter")
    if err != nil {
        return err
    }
    new := old + req.Delta
    if err := kv.Write(context.Background(), "counter", new); err != nil {
        return err
    }

    type response struct {
        Type string `json:"type"`
    }

    return n.Reply(msg, response{Type: "add_ok"})
})

And when a read request arrives:

n.Handle("read", func(msg maelstrom.Message) error {
    value, err := kv.ReadInt(context.Background(), "counter")
    if err != nil {
        return err
    }

    type response struct {
        Type  string `json:"type"`
        Value int    `json:"value"`
    }

    return n.Reply(msg, response{Type: "read_ok", Value: value})
})

Well, if you’re experienced with this kind of thing, you probably know this won’t work. And that is really the case. But it’s a good starting point and a way of validating that Maelstrom checks are solid. If you’re not experienced, the reason this does not work is that the read-modify-write inside add is being executed by concurrent nodes. If two nodes both read counter = 5 at the same time, they’ll both write 5 + delta, and one write will be lost.

There are two solutions to this problem: one is making that operation atomic, and the other is by making sure the node’s writes don’t conflict with one another. In the latter approach, we enter into the CRDT world.

Solving read-modify-write with CAS

We saw that our key-value store offers a CompareAndSwap method. With that, we can make our operation atomic:

for {
    old, _ := kv.ReadInt(context.Background(), "counter")
    if err := kv.CompareAndSwap(context.Background(), "counter", old, old+req.Delta, true); err != nil {
        // CAS failed retry
        continue
    }
    break
}

I’m being a bit sloppy here with this infinite loop. Ideally, we’d have a timeout and would also check for specific CompareAndSwap errors, so we don’t retry all of them. But you get the idea.

If you run the test, you may get a valid result. If you’re not really that curious to understand why you got a valid result, you’ll probably move on to the next challenge and miss many extra learnings. And that is because you may also get an invalid result for this solution. And that’s where the weirdness starts. This solution is not deterministic.

Before exploring that weirdness, I’d like to discuss an alternative solution that touches a bit on what a CRDT looks like.

CRDT-like solution

There are different kinds of CRDTs, and G-Counter is one of them. If we look at the mathematical definition of the G-Counter CRDT on Wikipedia (link), you’ll see:

payload integer[n] P
    initial [0,0,...,0]

update increment()
    let g = myId()
    P[g] := P[g] + 1

query value() : integer v
    let v = Σi P[i]

compare (X, Y) : boolean b
    let b = (∀i ∈ [0, n - 1] : X.P[i] ≤ Y.P[i])

merge (X, Y) : payload Z
    let ∀i ∈ [0, n - 1] : Z.P[i] = max(X.P[i], Y.P[i])

Let’s ignore compare and merge for a moment. If we squint and think a bit, the payload integer, update increment and query value kind of map to the responsibilities of our node: init, add, and read.

What is roughly being said in there is that we’ll have as an initial state a vector [0,0,...,0]. When a node needs to increment the counter, it only updates its counter (Wikipedia shows +1; we generalize to +delta). See the myId, and the P[g] := P[g] + 1 operation? And when we read the counter value, we actually sum all values of the vector. Interesting.

Let’s map this to what we have. Our state is stored on a key-value store, and it does not offer a vector data structure. However, if a node is only updating its part of the vector, we can create a key-value counter for each node and consider that to be a vector. Using the node ID as the key, our init becomes:

n.Handle("init", func(msg maelstrom.Message) error {
    return kv.Write(context.Background(), n.ID(), 0)
})

Our read-modify-write becomes:

old, err := kv.ReadInt(context.Background(), n.ID())
if err != nil {
    return err
}
new := old + req.Delta
if err := kv.Write(context.Background(), n.ID(), new); err != nil {
    return err
}

Our counter is now spread into different counters, and the solution we saw on Wikipedia says that we should sum them to get the real value. Let’s do that:

var sum = 0
for _, nodeID := range n.NodeIDs() {
    value, err := kv.ReadInt(context.Background(), nodeID)
    if err != nil {
        return err
    }
    sum += value
}

And, as before, if we run the test, you may get a valid or invalid result.

So, what happened here is that by making each node work on its counter, they don’t conflict with one another when writing, eliminating the race condition we had.

So is this a CRDT? Not quite. The merge and compare pieces, part of the definition that we ignored, are what would make this a CRDT. But what are those? Well, CRDT is a decentralized algorithm. But the fact we’re using a centralized key-value store to build our solution means we have eliminated the need for merge and compare. If we were to remove the key-value store, we would need a way for nodes to know which values the other nodes have. Currently, it is the key-value store that is doing this job. A node can always read from the key-value store to know what the other node has; that’s how our read works. So, what happens in a real CRDT algorithm is that the key-value store would be replaced by a gossip algorithm used to broadcast local node state, and compare and merge used to reconcile with others.

I did not explore this approach on this challenge but wanted to share that to make the learnings from this more complete. In the future, I’ll see if I can use the approaches discussed on Fly.io’s Broadcast Challenges, add the merge and compare, and pass the test.

On consistency models

Sequential consistency

So, we discussed two solutions, but none of them deterministically pass the test. We need to figure out why and fix our code. The fact that the problem specification suggests the use of SeqKV gives us a hint where our issue may lie. And proposing its use may indicate something about the author’s pedagogical intentions.

SeqKV is a sequentially consistent key-value store. To understand what that means, we have to understand what a consistency model is, because sequential consistency is one of many. One way of thinking about the consistency model of a data storage is to think that there’s a contract between clients and the data storage. If clients agree with certain rules, the data storage promises that it will behave in a certain way. You might be thinking, what are these behaviors, and why doesn’t all data storage behave the same way? Well, in a distributed system with concurrent clients where nodes might fail, messages can get delayed, and clocks are unreliable, weird things can happen, and different kinds of behaviors emerge. We can call these odd behaviors anomalies. These anomalies were categorized, and a consistency model type is essentially defined by the set of anomalies that it allows to happen. The stronger the model, the more restrictive it is in terms of what kinds of anomalies it allows. And the reason every data storage doesn’t pick the stronger model when designing a system to achieve a certain consistency model is that there’s a compromise around availability and performance that you might not be willing to give up. Consistency Models is a good reference on the different consistency models.

Now we need to understand the behavior that a sequentially consistent key-value store promises its clients.

In a sequentially consistent model, a total order of all operations is required, and that must be consistent with each client’s program order. However, the total order of events may not be what in fact happened in reality. This is the anomaly it allows.

I’ll use Maelstrom’s example to explain further. Suppose two clients execute the following operations in real-time order:

client1: write x = 1
client2: CAS x (1 -> 2)
client1: write x = 1
client2: read x = 2

You might be thinking that there’s something wrong with the fact that client2 read x = 2. But that anomaly is totally valid under sequential consistency. Maybe the third operation got delayed, and client2 is doing a stale read. Or the CAS got delayed and happened after the second write. The following two orders are valid:

client1: write x = 1
client1: write x = 1
client2: CAS x (1 -> 2)
client2: read x = 2

client1: write x = 1
client2: CAS x (1 -> 2)
client2: read x = 2
client1: write x = 1

We now understand sequential consistency. Let’s see if that is enough to understand our results.

Does sequential consistency explain our results?

Here’s one of the results I got in one of my runs:

:workload {:valid? false,
    :errors (#jepsen.history.Op{:index 3938,
                                :time 30006761427,
                                :type :ok,
                                :process 0,
                                :f :read,
                                :value 1343,
                                :final? true}),
    :final-reads (1345 1343 1345),
    :acceptable ([1345 1345])},

Process 0 read 1343 and it was expected 1345. Okay, given our understanding of sequential consistency, it might be the case that, for this run, the final read happened before the last write, so we’re getting a stale value. Reasonable. It’s a valid anomaly. And for some runs the anomaly does not happen, and the test passes.

That kind of explains our results. But there’s something deeper going on. To realize there’s something deeper going on, we need to look at how the final results are checked. If we look at the Maelstrom source code of the test run, we’ll see the following pattern:

Run the main workload
Heal partitions
Sleep for 10 seconds. Waiting for recovery…
Send the final reads to all nodes

So, there’s a cooldown period of 10 seconds before the final reads happen. That is strange. Our read continues stale even after 10 seconds. You could be asking, how long do we have to wait for writes to converge?

I asked that question. And the answer to that leads to a very important realization about consistency models. Maelstrom is a platform built for learning purposes. So SeqKV was built as a way to illustrate what sequential consistency might look like in practice. It is a key-value store that simulates a sequential-consistency-ish behavior. But the thing is that sequential consistency only defines what anomalies are allowed, not what anomalies must occur and how they occur. Our final stale read that never seems to get the most recent value, although it’s a valid anomaly, is a behavior that has nothing to do with sequential consistency, in the sense that different databases, under the same consistency model, might have a different manifestation of that anomaly.

The hacky solution

So, the kind of unfortunate conclusion is that to solve the challenge, we really need to understand some implementation details of SeqKV. Just understanding sequential consistency is not enough.

By reading Compare-and-swap on seq-kv and SeqKV source code, you get a sense of what is going on. The two crucial parts: SeqKV has some randomness on reads allowing the node to pick state from history, and if a change of state occurs the client is forced to have the most up-to-date view of the data store.

And with that we can come up with a solution for g-counter that uses SeqKV. If clients are forced to have the most up-to-date view when a change of state occurs, we can simply do a write of a unique value before reads:

kv.Write(context.Background(), "rand", time.Now().UnixMilli())

Any read that comes after the write will not be stale anymore. It is important that the value be unique. SeqKV only forces clients to the latest state when the write actually changes state. If the value matches what’s already there, nothing changes and the reads stay stale.

The “write forces clients to the latest state” behavior is just how Maelstrom’s SeqKV works internally to keep the total order well-defined. Do not take this additional write as a workaround for the anomalies of any sequentially consistent database. You’re not fixing the anomaly. There’s not really any fixing to be done. Your data store operates under a certain consistency model, and you have to deal with that. Of course, in this case, by knowing how SeqKV works internally, we could make a change to our program to behave the way we want it to behave. That’s why it can feel hacky.

The non-hacky solutions

If the hack bothers you, there are a couple of approaches to the g-counter workload. The more appropriate solution is to implement a CRDT with the gossip mechanism as we discussed. A G-Counter is a CRDT, and the workload is made for that. But if you’re willing to explore the consistency model path, you can switch from SeqKV to LinKV to get a linearizable key-value store. A linearizable key-value store operates under the linearizable consistency model. Now the total order observed by clients must match real-time order, and we don’t get stale reads. And the tests pass deterministically. If you’re wondering what is required to implement linearizability, you may be able to use the solution we discussed in Generating Unique IDs with Raft Consensus to add consensus on the nodes of your G-Counter. Exploring alternatives makes it easier to understand the trade-offs.

This challenge was brutal for me. Not in terms of coding. But it took a while to understand all the nuances of what was going on. In any case, it was fun, and I learned a lot. That’s all for this one.

Solution can be found at maelstrom-g-counter.

Fly.io's Broadcast Challenges

2026-04-09T18:33:59+00:00

This post is part of a series on Fly.io’s distributed systems challenges:

Implementing Snowflake Unique ID Generation

Generating Unique IDs with Raft Consensus

Fly.io’s Broadcast Challenges (this post)

Building a Grow-Only Counter on a Sequentially Consistent KV Store

This is another post about the series of distributed systems challenges by Fly.io. I’ve talked about using Snowflake and Raft consensus to solve Challenge #2: Unique ID Generation, in Implementing Snowflake Unique ID Generation and Generating Unique IDs with Raft Consensus. Now we’re going to discuss a set of Broadcast challenges that start at Challenge #3a: Single-Node Broadcast.

In this set, we are given the task of implementing a gossip algorithm that is fault-tolerant and efficient. We start at a simple single-node implementation, move towards a multi-node one, then implement fault tolerance, and last, try to improve its efficiency. It is really nice how the challenges are set up and how you can build on top of the previous solution.

Single-node

There’s not much to discuss here. The solution is straightforward: receive the message, record it, and send broadcast_ok back. The only thing to be aware of is to make sure there are no duplicates.

code: single-node

Multi-node

Keeping it simple is enough for this challenge. We can simply broadcast any new message to every neighbor we know. The interesting thing about this challenge is the fact that only new messages can be sent. If that check is not done, messages will be in a loop, hopping from node to node, congesting the network. This hints to us at the importance of trying to keep the network congestion low.

This algorithm is enough to pass the challenge, but what happens if we add network partition?

code: multi-node

Fault tolerance

Network partitions are introduced. It means that for a brief period of time, some nodes will not be available. We need some kind of mechanism to make sure messages always make their way through all nodes eventually, after the partition is healed.

Two possible approaches:

Add a retry queue for timed-out messages

When you try to send a message to a neighbor and it times out, you add that message to a retry queue. Then a background job that listens to the queue periodically sends those messages. The messages only leave the queue when the neighbor acks. There are multiple flavors of this approach, actually. What is important is the strategy: make sure every new message you receive is acknowledged by your neighbors. If we can make sure every message sent is acknowledged, all nodes will eventually have all messages.
Periodic anti-entropy

We don’t have to immediately forward the new messages. An alternative to that: nodes can periodically exchange messages with their neighbors, detect missing entries, and fill the gaps. Because nodes are periodically running this repair mechanism, eventually, everyone catches up.

For this challenge I decided on using the first approach. It is not a very robust retry queue with adequate retry intervals, but it is good enough for us.

retries := make(chan RetryMessage, 100)

// have 5 background jobs periodically retrying messages
for i := 0; i < 5; i++ {
    go func() {
        for job := range retries {
            type req struct {
                Type    string `json:"type"`
                Message int    `json:"message"`
            }
            r := req{Type: "broadcast", Message: job.message}

            ctx, cancel := context.WithTimeout(context.Background(), RPCTimeout)
            _, err := n.SyncRPC(ctx, job.destination, r)
            cancel()
            if err != nil {
                job := job
                time.AfterFunc(RetryInterval, func() {
                    retries <- job
                })
            }
        }
    }()
}

We create 5 goroutines that listen to the retry queue and resend failed messages. If it fails again, it re-enqueues. The issue with this approach is the infinite retry. Having an infinite retry assumes nodes can’t crash and topology is stable, which is not true in real life. If we add a limit, we violate the broadcast guarantee that all nodes eventually see all messages. So, adding anti-entropy to the mix is probably a good idea, and something I definitely would like to explore in the future.

code: fault-tolerant

Efficiency

When we’re building an efficient broadcast algorithm, we’re usually trying to optimize for low latency and low messages per operation.

Latency is how long it takes for a message to be seen in all nodes. An algorithm that is fast to converge is desirable.
Messages per operation is the average number of network messages generated per broadcast operation. The lower the msgs/op is, the less congested the network is, meaning a lower amount of resources is needed by the node. That is a more scalable solution.

Essentially there are two major things that impact them that need further investigation: topology and batching.

Topology

Let’s look at the following 7-node topology:

Every node is connected to every node. What happens when a new message arrives at n0? It broadcasts it to all other 6 nodes. And when that message arrives at n1, it will broadcast the same message to all other nodes (except n0). But that was unnecessary, because n0 already sent the message to the other nodes. There’s some redundancy going on, generating more messages than required.

In contrast to that, let’s look at the other topology:

When a message arrives at n0, it sends to n1 and n2, then n1 sends to n3 and n4, and n2 sends n5 and n6. No redundancy.

What is happening here is that in the first topology, there are multiple paths between any pair of nodes. So a message will travel through all of those paths. In the second one, there’s only one path between any pair of nodes, meaning there’s no way for a node to ever receive the same message from a different path. The second topology is a graph with no cycles, that is, a tree.

From that we can conclude that a tree is the minimum needed to reach everyone, and adding extra edges will only increase msgs/op. So, we should aim to work with a topology as close as possible to a tree.

Within a tree topology, there are multiple configurations. For example, at one extreme we have the line topology:

An operation at node n0 will hop from node to node until it reaches the last one. It takes quite a while for an operation to travel through the network.

At the other extreme, we have the star topology:

It is natural to see that the latency in the line topology will be higher than in the star topology. And the balanced tree is a middle ground.

The branching factor of the tree also impacts msgs/op. Imagine a network partition in n1. In the line topology, a message from n0 will not reach the other nodes, and it will need to be retried until the partition heals. That does not happen in the star topology. In the star topology, a partition at n1 only blocks n1, n0 still reaches everyone else directly, so fewer retries are needed.

Batching

The second thing to look into is batching. Instead of generating one message per operation, we can batch multiple broadcasts into a single message, decreasing msgs/op.

But you can see that to introduce a batching mechanism, we need to necessarily introduce some latency. The node had to wait a bit for a batch of broadcasts to arrive first. So there’s a trade-off between msgs/op and latency.

To implement batching, I’ve built on top of our fault-tolerant strategy. Now we keep track of all unacked messages per neighbor and periodically send all unacked messages in a single request. That required a new internal maelstrom message, that we called gossip.

Another thing that can be considered is whether we really need an immediate broadcast to the neighbors as we need a new message. By having a background routine, we can simply mark the new incoming messages as unacked and let the background routine deal with it. The decision of having or not having this immediate call also impacts msgs/op and latency.

With these two mechanisms in hand, topology and batching, we can tweak a couple of parameters to figure out reasonable values of latency and msgs/op that satisfy the requirements.

code: fault-tolerant-efficient-1 and fault-tolerant-efficient-2

Benchmark

To understand the trade-offs discussed above and figure out a reasonable topology and batch interval, I’ve run the same test with different batch intervals and topologies. You can see the batch interval used in the INTERVAL column, and the topology is defined by the branching factor of the tree in the BRANCH column.

Interval (ms)	Branch	msgs/op	p50	p95	p99
100	1	13.31	2281	3363	3646
100	2	10.70	867	1154	1214
100	4	9.95	512	704	795
100	8	9.33	416	587	599
100	16	9.28	348	438	457
200	1	8.80	3046	4397	4640
200	2	7.30	1155	1466	1567
200	4	7.02	708	955	981
200	8	6.74	566	749	791
200	16	6.74	414	608	636
300	1	7.26	4365	6580	6775
300	2	6.33	1667	2114	2222
300	4	5.91	958	1249	1331
300	8	5.80	723	981	1056
300	16	5.89	484	711	780
500	1	6.21	6661	10857	11287
500	2	5.47	2733	3467	3584
500	4	5.30	1648	2029	2143
500	8	5.24	1170	1532	1655
500	16	5.24	794	1112	1177

The test used was:

./maelstrom test -w broadcast \
    --bin ~/go/bin/fault-tolerant-efficient-2 \
    --node-count 25 \
    --time-limit 20 \
    --rate 100 \
    --latency 100

Nice to see that our understanding about trade-offs checks out: higher branching implies lower latency, and, at the same branching, higher interval implies lower msgs/op but higher latency. Also, we can see diminishing returns on branching.

Back to the challenge. Given the requirements of:

Messages-per-operation is below 20
Median latency is below 1 second
Maximum latency is below 2 seconds

We see that 11 out of 20 configurations pass the test. Awesome. We would not be able to achieve that without a tree topology and a batching strategy.

And that’s all I got for this one.

Generating Unique IDs with Raft Consensus

2026-04-08T18:15:13+00:00

This post is part of a series on Fly.io’s distributed systems challenges:

Implementing Snowflake Unique ID Generation

Generating Unique IDs with Raft Consensus (this post)

Fly.io’s Broadcast Challenges

Building a Grow-Only Counter on a Sequentially Consistent KV Store

This blog post is a follow-up on Implementing Snowflake Unique ID Generation. In that post, I explain an implementation of Snowflake IDs that can be used to solve the distributed systems challenge Challenge #2: Unique ID Generation.

Now, we discuss an alternative to that. We’re going to use Raft to generate globally unique IDs. We assume here a general understanding of how Raft works. I recommend The Secret Lives of Data, just in case. The idea is to see if we can use the etcd-io/raft library to build a program that passes the challenge.

Here’s the full code we’ll discuss: maelstrom-unique-ids-raft.

Thinking about the system architecture

One challenge in building a Raft application is the architecture of the code. Because you have to wait for nodes to achieve consensus, it can be a bit confusing how to architect that waiting. The architecture also depends on the API provided by the Raft library.

Here’s an idea on how we can architect the code to build a distributed counter using an embedded Raft library that communicates with maelstrom clients. In essence, we can do something very similar to what we’ve done with Snowflake. Have some kind of object, and call a method (e.g. Next) on that object to get the next number in the sequence whenever a generate message from maelstrom comes in.

We’ll call this object DistributedCounter.

type Result int

type DistributedCounter struct {
    // state under consensus
    seq int

    // deps
    node    raft.Node
    storage *raft.MemoryStorage

    // ...
}

func (dc *DistributedCounter) Next(ctx context.Context, reqID string) (Result, error) {

    // request/propose a new value to raft nodes and wait for it
}

This object will act as a Raft node, communicate with other Raft nodes to achieve consensus, and store the Raft log. That’s why we see a raft.Node and *raft.MemoryStorage as dependencies. Also, seq is the state for our globally unique ID, and it is incremented whenever consensus is achieved.

And this is how everything fits together:

// in main.go

n := maelstrom.NewNode()

var counter *DistributedCounter

n.Handle("init", func(msg maelstrom.Message) error {
    // initialize raft nodes and distributed counter

    return nil
})

n.Handle("generate", func(msg maelstrom.Message) error {
    next, err := counter.Next(context.Background(), generateRequestID(16))

    // ...

})

You can see the similarities with the Snowflake solution. There, we had a Generator, that needed to be initialized inside init, with a Next method. Here we have a DistributedCounter, that needs to be initialized in init, with a Next method. In essence, we need to implement these 3 methods: the init handler, the generate handler, and the Next method. There’s more to it, actually, but we’ll get there.

Generate implementation

Starting with generate because it’s the simplest and there’s no Raft explicitly involved (we treat it as a black box for now). We can simply return the value we got from our counter back to maelstrom client.

n.Handle("generate", func(msg maelstrom.Message) error {
    next, err := counter.Next(context.Background(), generateRequestID(16))
    if err != nil {
        return maelstrom.NewRPCError(maelstrom.TemporarilyUnavailable, err.Error())
    }

    type response struct {
        Type string `json:"type"`
        Id   string `json:"id"`
    }

    return n.Reply(msg, response{
        Type: "generate_ok",
        Id:   fmt.Sprintf("%d", next),
    })
})

We add a “random” request ID for every generate request, because in our DistributedCounter implementation, when a node proposes a next ID, it needs to wait for the nodes to achieve consensus before getting the next ID. And that waiting happens through a channel. So we use a request ID to map results to requests. Whenever a result is ready, we associate it with its request, and the node waiting on that request ID can respond to its client.

Getting the next value

Next can be implemented simply using Propose from raft.Node.

func (dc *DistributedCounter) Next(ctx context.Context, reqID string) (Result, error) {
    ch := make(chan Result, 1)
    dc.mu.Lock()
    dc.waiters[reqID] = ch
    dc.mu.Unlock()

    // If this node is a leader, it will append the entry and send to the followers waiting for majority
    // If not, the proposal is sent internally to the leader
    // We retry until raft accepts the proposal (e.g., no leader yet).    
    for {
        if err := dc.node.Propose(ctx, []byte(reqID)); err == nil {
            break
        }
        time.Sleep(100 * time.Millisecond)
    }

    // blocked waiting for a result
    return <-ch, nil
}

A couple of things to discuss here. We add a waiters map[string]chan Result to our object. For every request, we create a channel, and our node waits on that channel:

// blocked waiting for a result
return <-ch, nil

To get the next number in the sequence, we need to propose an increment to our cluster, hence node.Propose. The Raft protocol is agnostic in regard to the data it replicates. It is just concerned about making sure nodes execute the same operations in the same order. In our case, there’s only one operation and it is a simple increment (e.g., seq++). In more complex applications, it can be a command, for example, set x = 2, or a SQL statement. I’m saying this because, given the simplicity of our application, there’s not much we need to propose to our nodes; we just need to signal them that a new ID was requested. And they will achieve consensus on that signaling. So that’s why we just send a reqID. Whenever consensus is achieved, nodes will use this value to associate the new ID with reqID using the waiters map.

The other thing to say is how Propose works. In Raft, only the leader can propose a new entry to the log. The recommended approach in the case of a proposal request arriving at a follower is to deny the request and send the information about the leader back to the client. If we do that, our maelstrom test will fail, because maelstrom clients are not expecting that kind of interaction. However, Propose detects internally whether the node is the leader, and if not, it sends the proposal to the leader. So we don’t have to worry about that.

One last thing. The proposal can fail. Maybe the network is going through a partition and is having a hard time electing a new leader, so no new proposals can be made. In this case, we naively retry until it works. A more appropriate solution would be to add a timeout and return an error to the client. I’ll come back to this later.

Initializing our distributed counter

There are two parts to init: we need to initialize the Raft nodes, and we need to start a background routine in our counter that runs the logic the Raft library expects our application to execute.

Initializing Raft nodes

Here, we need to figure out the node ID and all the peers and start the Raft node with the right configuration.

nodeNum, err := strconv.ParseUint(n.ID()[1:], 10, 64)
if err != nil {
    return fmt.Errorf("failed to parse node id: %v", err)
}
raftID := nodeNum + 1

var peers []raft.Peer
for _, nid := range n.NodeIDs() {
    num, err := strconv.ParseUint(nid[1:], 10, 64)
    if err != nil {
        return fmt.Errorf("failed to parse peer id: %v", err)
    }
    peers = append(peers, raft.Peer{ID: num + 1})
}

storage := raft.NewMemoryStorage()
cfg := &raft.Config{
    ID:              raftID,
    ElectionTick:    10,
    HeartbeatTick:   1,
    Storage:         storage,
    MaxSizePerMsg:   4096, 
    MaxInflightMsgs: 256,
}

raftNode := raft.StartNode(cfg, peers)
counter = NewDistributedCounter(raftNode, storage, n.Send)
go counter.run()

For this application, we’re using a memory storage, but we can swap that if we want. A HeartbeatTick of 1, meaning every clock tick, a heartbeat is sent. And ElectionTick of 10: if nodes don’t hear from leader in 10 clock ticks, a leader election starts.

Background routine

The background routine is what runs when we call go counter.run(). If we look at etcd-io/raft Readme, it says that the Raft client application has certain responsibilities. Let’s start with the 2 high-level responsibilities: indicating how frequently the clock ticks, and being ready to act on a batch of updates:

ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop()

for {
    select {
    case <-ticker.C:
        dc.node.Tick()

    case rd := <-dc.node.Ready():
        // ...
    }
}

Now we know that heartbeats will happen every 100ms and the election timeout is 1s. And node.Ready is a channel that informs us that a new batch of updates is available, and the application needs to act on it. Inside Ready, there’s a bunch more responsibilities. In our case, there are five:

// 1. store updated term/vote/commit
if !raft.IsEmptyHardState(rd.HardState) {
    if err := dc.storage.SetHardState(rd.HardState); err != nil {
        log.Printf("failed to set hard state: %v", err)
    }
}

// 2. append log entries to local storage
if len(rd.Entries) > 0 {
    if err := dc.storage.Append(rd.Entries); err != nil {
        log.Printf("failed to append entries: %v", err)
    }
}

// 3. if we have new messages to send to other nodes, send it
for _, msg := range rd.Messages {
    dc.send(msg)
}

// 4. if we have log entries that have achieved consensus 
//    apply your application logic
for _, ent := range rd.CommittedEntries {
    dc.applyEntry(ent)
}

// 5. indicates that we're done
dc.node.Advance()

This is what the Raft library provides as an API so we can make our application work with the consensus engine it uses. I wasn’t familiar with this library, so it was a bit hard to grasp at first. But if you’re familiar with Raft, you can get a sense of what is going on. Other libraries will probably offer a different API. I won’t spend too much time on the details of each of these responsibilities. The most important for our discussion are 3 and 4. We’ll skip 3 for now and focus on 4.

So, what happens in 4 is that we have received all new log entries that achieved consensus. We’re in a state where the Raft log is replicated across nodes, and our application’s state needs to reflect that. The library is telling you to apply these new entries to your application so that its state is consistent across nodes. If the application is a decentralized database, the entry could be a SQL statement, for example. In our case, it is just a request ID signaling that the sequence must be incremented.

Here’s applyEntry implementation:

func (dc *DistributedCounter) applyEntry(ent raftpb.Entry) {
    if ent.Type == raftpb.EntryConfChange {
        var cc raftpb.ConfChange
        if err := cc.Unmarshal(ent.Data); err != nil {
            log.Printf("failed to unmarshal conf change: %v", err)
            return
        }
        dc.node.ApplyConfChange(cc)
        return
    }

    if ent.Type != raftpb.EntryNormal || len(ent.Data) == 0 {
        return
    }

    reqID := string(ent.Data)

    dc.mu.Lock()
    dc.seq++
    res := Result(dc.seq)

    ch := dc.waiters[reqID]
    if ch != nil {
        delete(dc.waiters, reqID)
    }
    dc.mu.Unlock()

    if ch != nil {
        ch <- res
    }
}

It starts with some more responsibilities that I’ll ignore. The important thing is that it is here that seq is incremented and the value is put on waiters[reqID] so that the node waiting for that value can respond to the client. There’s a very subtle thing happening here that might not be obvious:

if ch != nil: this is the node that proposed a new ID and is waiting on another blocked goroutine for that ID
if ch == nil: this entry was proposed by a different node, so we don’t have to send the value to the channel

Note that all nodes increment the sequence to ensure the value is consistent across the cluster.

Communication among the nodes

We’ve implemented the core of our application. But there’s one supporting piece missing. Nodes need to be able to communicate with each other. We said that if a follower receives a proposal, it needs to send it to the leader. Also, the leader needs to send entries to followers. There are a bunch of messages being exchanged, and we did not set that up. That’s where that third responsibility above comes in:

for _, msg := range rd.Messages {
    dc.send(msg)
}

Whenever the protocol indicates that new messages need to be sent to other nodes, we need to send them. Here’s how we implement that send, hooking that up with maelstrom :

func (dc *DistributedCounter) send(msg raftpb.Message) {
    destID := fmt.Sprintf("n%d", msg.To-1)

    data, err := msg.Marshal()
    if err != nil {
        log.Printf("failed to marshal raft message: %v", err)
        return
    }

    if err := dc.sendFn(destID, map[string]any{
        "type": "raft",
        "data": data,
    }); err != nil {
        log.Printf("failed to send raft message to %s: %v", destID, err)
    }
}

And sendFn is a dependency that we add in DistributedCounter

type DistributedCounter struct {
    // state under consensus
    seq int

    // deps
    sendFn  func(dest string, body any) error
    node    raft.Node
    storage *raft.MemoryStorage

    //control
    mu      sync.Mutex
    waiters map[string]chan Result
}

// in main.go

counter = NewDistributedCounter(raftNode, storage, n.Send)

You see that we created a new kind of maelstrom message called raft. We need to handle that as well:

// maelstrom handler
n.Handle("raft", func(msg maelstrom.Message) error {
    var body struct {
        Data []byte `json:"data"`
    }
    if err := json.Unmarshal(msg.Body, &body); err != nil {
        log.Printf("failed to unmarshal raft envelope: %v", err)
        return nil
    }

    if err := counter.Sync(context.Background(), body.Data); err != nil {
        log.Printf("failed to sync raft: %v", err)
    }
    return nil
})

// a new method on DistributedCounter
func (dc *DistributedCounter) Sync(ctx context.Context, data []byte) error {
    var msg raftpb.Message
    if err := msg.Unmarshal(data); err != nil {
        return err
    }
    return dc.node.Step(ctx, msg)
}

We can see that whenever a new internal message is received, we eventually call node.Step. That is the method that the library provides to be called whenever new messages arrive.

And with that, our solution is completed.

Running the test

So I ran the same test I ran for Snowflake:

./maelstrom test -w unique-ids \
    --bin ~/go/bin/maelstrom-unique-ids-raft \
    --node-count 5 \
    --time-limit 30 \
    --rate 1000 \
    --availability total \
    --concurrency 10 \
    --nemesis partition

And the test failed. All that work to get a failing test!

But if we think about it and look at the test we’re running, it kind of makes sense. We’re saying we want total availability even with a network partition. However, Raft is not a totally available system. Depending on how the network partition occurs, the system may become unavailable while it tries to elect a new leader. We can either reduce our availability requirement or remove the requirement that it needs to be fault-tolerant to network partitions. The following tests pass:

# availability 0.999
./maelstrom test -w unique-ids \
    --bin ~/go/bin/maelstrom-unique-ids-raft \
    --node-count 5 \
    --time-limit 30 \
    --rate 1000 \
    --availability 0.999 \
    --concurrency 10 \
    --nemesis partition

# no network partition
./maelstrom test -w unique-ids \
    --bin ~/go/bin/maelstrom-unique-ids-raft \
    --node-count 5 \
    --time-limit 30 \
    --rate 1000 \
    --availability total \
    --concurrency 10 

That is evidence of Raft’s CP (from the CAP theorem) nature and the trade-offs between availability and network partition tolerance.

One thing that bugged me was that the application was implemented so that Next should never fail. Remember that ugly retry loop? So why don’t we get 100% availability? Turns out maelstrom has a 5-second timeout on the client. If the client doesn’t hear a response within 5s, it will count as a failure. By default, partitions last 10s in tests. So that makes sense. But even if you set the partitions to last 2s, you still don’t get 100% availability. My hypothesis is that there’s probably a network partition at the end of the test run, so requests made close to the end never get a response, and those failures are counted.

Comparing with Snowflake

It is interesting to compare the consensus approach with the Snowflake approach. You clearly see the trade-offs of the CAP theorem in action. The Snowflake approach is totally available even with network partitions. Meaning it is compromising on consistency somehow. That makes sense. You cannot get an orderly increasing sequence with no gaps (e.g., 1, 2, 3, 4, ….) with Snowflake. Snowflake IDs are only roughly sortable, not strictly ordered.

Another comparison is the amount of overhead coordination adds. The table below compares the latency of both approaches.

Metric	Raft	Snowflake
Ops	23,269	27,965
msgs/op	11.2	0.0
Min	0.3 ms	0.1 ms
Mean	3.7 ms	0.4 ms
Median	0.7 ms	0.4 ms
p95	2.4 ms	0.8 ms
p99	5.5 ms	1.1 ms
Max	4,250 ms	10.6 ms

I mean, this is all running locally and does not reflect real-world scenarios, but it is interesting to see the latency increase, the effect of the network partition on max, and the introduction of internal messages being reflected on msgs/op.

That was a fun challenge. I hope you have enjoyed reading as much as I have enjoyed writing. And that’s all for this blog post.

Implementing Snowflake Unique ID Generation

2026-04-07T14:59:10+00:00

This post is part of a series on Fly.io’s distributed systems challenges:

Implementing Snowflake Unique ID Generation (this post)

Generating Unique IDs with Raft Consensus

Fly.io’s Broadcast Challenges

Building a Grow-Only Counter on a Sequentially Consistent KV Store

I’ve wanted for some time to work on this series of distributed systems challenges by Fly.io, and I have finally found some time. This post is about Challenge #2: Unique ID Generation.

A brief explanation of what these challenges are all about for those not familiar. In essence, you are given a task to build a decentralized algorithm that should pass a test. You use maelstrom as the platform that provides the workload and checks if the results generated by your algorithm are valid for that workload. To give you a sense of how that works, here’s the command I’ve run for project 2:

./maelstrom test -w unique-ids \
    --bin ~/go/bin/maelstrom-unique-ids \
    --node-count 5 \
    --time-limit 30 \
    --rate 1000 \
    --availability total \
    --concurrency 10 \
    --nemesis partition

You run a maelstrom test specifying the workload, in this case unique-ids. You provide your task as a binary, e.g. ~/go/bin/maelstrom-unique-ids. You can set some parameters that influence how the test is run:

node-count: the number of processes maelstrom will spawn of your binary
time-limit: for how long the test runs
rate: requests/s
concurrency: clients running concurrently
nemesis: inject a fault, in this case a network partition
availability: the availability target that will be checked

Generating unique ids is a common problem in distributed systems and it is not particularly hard to solve, and for this challenge you can get a valid result pretty easily with solutions such as:

Use the node id as the namespace, and a sequence counter per namespace
Generate an UUID or ULID
Use a centralized database as the provider of the sequence

There are two approaches I wanted to explore for fun: implementing Twitter Snowflake and using a consensus algorithm. I mean, the idea is not just to pass the challenge but learn a bit more about the problem, alternative solutions, and trade-offs. In this post, I focus more on Snowflake, and hopefully I’ll write another one about using consensus.

It is kind of interesting to see their motivation in that blog post. They were using Cassandra, which had no built-in way of generating unique ids, and they needed an id that was roughly sortable, highly available, and had to fit into 64 bits. Interestingly enough, none of the solutions we listed above worked for them.

The solution was the following composition:

You have a sequence scoped by node id, and timestamp. The algorithm runs as follows:

Every time a new id is requested on a node (e.g. node 0), it grabs the current timestamp, e.g. 1775566509000, if this timestamp is different from the last timestamp used when an id was generated the sequence is reset.

The timestamp in this algorithm is the number of milliseconds since a custom epoch. I am using 1735689600000 as the custom epoch, so that’s why we use 39876909000 = 1775566509000 - 1735689600000

Also, 41 bits of timestamp is roughly 69 years. That means, Twitter has 69 years after their chosen epoch time before this algorithm expires.
If a new id is requested at the same timestamp, sequence is increased. So sequence is only increased when two requests happen at the same millisecond. Meaning we can have at most 4096 (12 bits) requests per millisecond in that node.
If it happens to be the case that we have more than 4095 requests, and sequence overflows, the algorithm waits for the next millisecond and resets the sequence.

So, I implemented this algorithm for the challenge.

package main

import (
    "sync"
)

const (
    nodeBits = 10
    seqBits  = 12

    maxNodeID = (1 << nodeBits) - 1
    maxSeq    = (1 << seqBits) - 1

    nodeShift = seqBits
    timeShift = seqBits + nodeBits
)

// 2025-01-01T00:00:00Z
const customEpoch int64 = 1735689600000

type Generator struct {
    mu sync.Mutex

    lastMs int64
    seq    int64
    nodeID int64

    clock Clock
}

func NewSnowflakeGenerator(nodeID int64, clock Clock) *Generator {
    return &Generator{
        lastMs: -1,
        seq:    0,
        nodeID: nodeID,
        clock:  clock,
    }
}

func (g *Generator) Next() int64 {
    g.mu.Lock()
    defer g.mu.Unlock()

    now := g.clock.Now().UnixMilli()

    if now == g.lastMs {
        if g.seq < maxSeq {
            g.seq++
        } else {
            now = WaitNextMillis(g.clock, g.lastMs)
            g.lastMs = now
            g.seq = 0
        }
    } else {
        g.lastMs = now
        g.seq = 0
    }

    ts := int64(g.lastMs - customEpoch)

    return (ts << timeShift) |
        (g.nodeID << nodeShift) |
        g.seq
}

And with that I was able to pass the challenge just fine. You can check the full solution to the challenge here.

When Does C++ Call the Move Constructor?

2025-09-04T22:35:40+00:00

This is a note to myself on some of the occasions the move constructor is called in C++. Let’s use a simple class that implements move constructor and move assignment with some print statements for debugging/confirmation.

class A {
public:
  A() { std::cout << "Default constructor called\n"; }

  A(A &&other) noexcept { std::cout << "Move constructor called\n"; }

  auto operator=(A &&other) noexcept -> A & {
    std::cout << "Move assignment called\n";
    return *this;
  }
};

I explore 5 cases, which I think cover most of the situations. I’d love to know if there are more.

1st Case - Explicit

The move constructor is called when we explicitly pass an rvalue to the class constructor:

A a; // Default constructor called
A a2(std::move(a)); // Move constructor called

2nd Case - Initializing from a temporary object

When we initialize a variable from an rvalue, the move constructor is called.

auto a3 = std::move(a2); // Move constructor called

Note that assigning to a variable already initialized does not call the move constructor, but move assignment instead:

A a4;
a4 = std::move(a3); // Move assignment called

3rd Case - Returning a local object from a function

This is an interesting one. When we have a function that returns a local object, for example:

A make_a() {
  A a;
  return a;
}

The move constructor can or cannot be called depending on whether the Return Value Optimization is applied or not. For this case, in C++17, the move constructor will not be called because the optimization is applied. We can disable RVO with the flag -fno-elide-constructors, and see the move constructor being called.

4th Case - Passing `rvalue` to a function

Here we have a function that receives A by value, meaning a new object A must be constructed. However, if you pass an rvalue to that function, you get a move constructor call.

void receive_a(A a) {}
receive_a(std::move(a5)); // Move constructor called

Think of this as the same as 2nd Case:

A a = std::move(a5);

Think about what happens in the following cases:

void receive_a(A&& a) {}

void receive_a(A&& a) {
    A new_a = std::move(a);
}

5th Case: Emplacing into standard containers

Standard containers usually implement move semantics and take ownership when an rvalue is passed:

std::vector<A> v;
A a6;
v.push_back(std::move(a6)); // Move constructor called

Scoped Threads in Rust

2025-09-01T21:17:25+00:00

Reading Two Beautiful Rust Programs I came across a construct I hadn’t seen before for working with threads in Rust: std::thread::scope.

When you create a thread in Rust, you must pass a closure of the code you want that thread to execute. Sometimes, that closure works on data from the parent scope (the code that is spawning the thread):

fn main() {
    let mut counter = 0;

    let f = || {
        counter += 1;
        println!("counter = {}", counter)
    };

    let _ = std::thread::spawn(f).join();
}

That code does not compile. This is essentially because std::thread::spawn takes a closure with the bound 'static, which means counter is technically able to outlive main. You cannot have a reference to something that might not be there (dangling pointer). So, you must force the closure to take ownership of counter. We fix that by adding move:

fn main() {
    let mut counter = 0;

    let f = move || {
        counter += 1;
        println!("counter = {}", counter)
    };

    let _ = std::thread::spawn(f).join();
}

What happens if we access counter at the end of the program?

fn main() {
    let mut counter = 0;

    let f = move || {
        counter += 1;
        println!("counter = {}", counter)
    };

    let _ = std::thread::spawn(f).join();

    println!("counter = {}", counter)
}

counter = 1
counter = 0

Because counter implements Copy, a copy of it is made and passed to the closure. What if we don’t want that behavior? Suppose we want to mutably borrow counter. We can use std::thread::scope to achieve this:

fn main() {
    let mut counter = 0;

    let f = || {
        counter += 1;
        println!("counter = {}", counter)
    };

    std::thread::scope(|s| {
        s.spawn(f);
    });

    println!("counter = {}", counter)
}

And that works:

counter = 1
counter = 1

Note that we don’t use move anymore. std::thread::scope defines a lifetime 'scope that is contained within the parent scope and lasts for the duration of the call to scope. It is guaranteed that all threads spawned within the scope will be joined before scope returns. Because threads cannot outlive 'scope, any data borrowed by their closures is guaranteed to remain valid for the lifetime of the threads. When you access counter again, there’s no mutable reference to it anymore.

Let’s use Box as an example of some heap-allocated data that does not implement Copy:

fn main() {
    let mut counter = Box::new(0);

    let f = move || {
        *counter += 1;
        println!("counter = {}", counter)
    };

    let _ = std::thread::spawn(f).join();
}

That works just fine. The problem is that if we want to access counter at the end of the program, we get an error:

println!("counter = {}", counter)

error[E0382]: borrow of moved value: `counter`
  --> src/main.rs:30:30
   |
20 |     let counter = Box::new(0);
   |         ------- move occurs because `counter` has type `Box<i32>`, which does not implement the `Copy` trait
21 |
22 |     let f = move || {
   |             ------- value moved into closure here
23 |         let mut c = counter;
   |                     ------- variable moved due to use in closure
...
30 |     println!("counter = {}", *counter)
   |                              ^^^^^^^^ value borrowed here after move

Our counter was moved and we don’t have access to it after the thread is finished. We can use std::thread::scope and mutably borrow the counter to fix this:

fn main() {
    let mut counter = Box::new(0);

    let f = || {
        let c: &mut i32 = &mut counter;
        *c += 1;
        println!("counter = {}", c)
    };

    std::thread::scope(|s| {
        s.spawn(f);
    });

    println!("counter = {}", counter)
}

There it is, a simple explanation of how to use std::thread::scope to execute a thread on borrowed data from the parent scope.

Writing a simple lexer in Rust

2023-09-20T00:00:00+00:00

Recently, I began to delve deeper into Rust. As a way to become more familiar with the language, I decided to write a simple lexer for a mathematical expression such as 10 - 3 + ( ( 4 / 2 ) * ( 8 * 4 ) ).

Writing a lexer shouldn’t be a difficult task, especially if you’ve built one in another language. I’ve tried to explore Rust features and write idiomatic code as much as possible, without getting too attached to past implementations in other languages.

If you’re not familiar with compiler theory, essentially a lexer is a phase of the compiler that transforms your text input into a list of meaningful tokens. Essentially, we want to implement a function like this:

fn tokenizer(input: String) -> Vec<Token> {
	...
}

In reality, lexers are not written in a way that produces all of the tokens at once. Usually they produce one token at a time, in an iterator fashion, and that token is passed to the parser that will check grammar rules and build an Abstract Syntax Tree.

We start by defining our tokens using an Enum:

enum Token {
    Number(i64),
    Plus,
    Dash,
    Star,
    Slash,
    LeftParen,
    RightParen,
    EOF,
}

For now, we will focus on the four basic operations, parentheses, and positive integers.

The typical approach to implementing a lexer is by iterating through the text and checking whether the current character is relevant for creating a token. This can be achieved using either a slice and indices or an iterator. In this task, I have chosen to use an iterator as it is considered the more idiomatic approach and allows me to explore the iterator-related APIs.

Essentially, the core pattern we need is:

let mut tokens: Vec<Token> = Vec::new();
let mut iter = input.chars();

while let Some(ch) = iter.next() {
    match ch {
    	// pattern matching logic
        // tokens.push(...)
    }
}

We loop through each character and apply pattern matching logic to it.

Since our mathematical expression is very simple, most of the characters are tokens themselves. Additionally, whitespaces have no meaning in our expression. Therefore:

match ch {
    ch if ch.is_whitespace() => continue,
    '(' => tokens.push(Token::LeftParen),
    ')' => tokens.push(Token::RightParen),
    '+' => tokens.push(Token::Plus),
    '-' => tokens.push(Token::Dash),
    '*' => tokens.push(Token::Star),
}

The only token that is slightly more complex is the Number, which is a sequence of characters. To implement it, we can utilize the matching ranges feature:

match ch {
    // ...
    '1'..='9' => {
        // ...
     }
}

To implement the body, I have decided to take a functional approach and use only iterators. The idea is to consume the iterator by extracting characters as long as they are digits. We can achieve this by using take_while:

match ch {
    // ...
    '1'..='9' => {
       let s = iter.take_while(|c| match c.is_ascii_digit()).collect();
     }
	
}

The issue with that approach is that s would not include the first digit, which is ch. Additionally, take_while would have already consumed the first non-digit character, resulting in the loss of one character in the subsequent iteration of the loop.

The way to work around this is by making iter Peekable and making use of next_if. With it we can peek at the next element and only consume it if it is a digit. Also, by combining it with from_fn we can create a new iterator chained with ch.

'1'..='9' => {
    let n: i64 = iter::once(ch)
        .chain(from_fn(|| iter.by_ref().next_if(|s| s.is_ascii_digit())))
        .collect::<String>()
        .parse()
        .unwrap();

     tokens.push(Token::Number(n));
}

With that we finish our tokenizer function:

fn tokenizer(input: String) -> Vec<Token> {
    let mut tokens: Vec<Token> = Vec::new();
    let mut iter = input.chars().peekable();

    while let Some(ch) = iter.next() {
        match ch {
            ch if ch.is_whitespace() => continue,
            '(' => tokens.push(Token::LeftParen),
            ')' => tokens.push(Token::RightParen),
            '+' => tokens.push(Token::Plus),
            '-' => tokens.push(Token::Dash),
            '*' => tokens.push(Token::Star),
            '1'..='9' => {
                let n: i64 = iter::once(ch)
                    .chain(from_fn(|| iter.by_ref().next_if(|s| s.is_ascii_digit())))
                    .collect::<String>()
                    .parse()
                    .unwrap();

                tokens.push(Token::Number(n));
            }
            _ => {
                panic!("unrecognized char");
            }
        }
    }

    tokens.push(Token::EOF);
    tokens
}

We can improve our error handling by defining a SyntaxError struct:

#[derive(Debug)]
struct SyntaxError {
    message: String,
}

impl SyntaxError {
    fn new(message: String) -> Self {
        SyntaxError {
            message
        }
    }
}

and returning Result<Vec<Token>, SyntaxError>

fn tokenizer(input: String) -> Result<Vec<Token>, SyntaxError> {
    let mut tokens: Vec<Token> = Vec::new();
    let mut iter = input.chars().peekable();

    while let Some(ch) = iter.next() {
        match ch {
            ch if ch.is_whitespace() => continue,
            '(' => tokens.push(Token::LeftParen),
            ')' => tokens.push(Token::RightParen),
            '+' => tokens.push(Token::Plus),
            '-' => tokens.push(Token::Dash),
            '*' => tokens.push(Token::Star),
            '1'..='9' => {
                let n: i64 = iter::once(ch)
                    .chain(from_fn(|| iter.by_ref().next_if(|s| s.is_ascii_digit())))
                    .collect::<String>()
                    .parse()
                    .unwrap();

                tokens.push(Token::Number(n));
            }
            _ => return Err(SyntaxError::new(format!("unrecognized character {}", ch))),
        }
    }

    tokens.push(Token::EOF);
    Ok(tokens)
}

It wasn’t a very hard exercise but sufficient for me to get familiarized with Rust’s syntax, its type system, some error handling patterns, iterator-related APIs, and pattern-matching syntax. I’ve also been doing Rustlings exercises on the side. For someone used to programming in Go, I’ve often found reading Rust code to be challenging. However, I’m gradually beginning to appreciate its elegance.

Future posts ideas: work on an iterator-based version of the lexer; build a parser to evaluate the expression; build a lexer for a more complex language.

The day I discovered vmtouch

2023-01-27T00:00:00+00:00

Introduction

Last weekend I decided to take a deeper look at the famous SQLite35% Faster Than The Filesystem benchmark. I didn’t want to do a shallow read of the post. I wanted to compile the kvtest tool and run the experiments myself and see what is going on. I recommend doing that, especially if you want to follow along with the blog post.

While running the read experiments, something caught my attention. It seems that to see good performance results using SQLite, you have to run kvtest twice discarding the first run results. So the first run is used to load data into the cache, and the second and following runs take advantage of that. That makes sense but for some reason that intrigued me a bit and I became curious to understand more about what was happening between the first and second run.

The first question that popped into my mind was: what kind of caching is happening here? I understand that SQLite has its buffer pool implementation that caches pages into memory. But, it’s clear that the performance does not come from that since the kvtest program is completely finished after the first run. In the second run, the buffer would start empty (I suppose). So, the only place left for caching is in the Operating System. That reasoning gained more credibility when I saw the Figure 1 image from SQLite Database System: Design and Implementation book: To confirm that I wanted to tinker with the disk page cache. In the search to figure out how I could pin or evict pages from the cache, I ended up, founding an awesome tool called vmtouch.

Confirming that the OS is caching the `SQLite` data

Let’s first purge the OS page cache: sudo purge (in MacOs, in Linux it seems that echo 3 > /proc/sys/vm/drop_caches does the trick)
Calling vmtouch test1.db tells that 0% of test1.db is cached

>> vmtouch test1.db                             
Files: 1
Directories: 0
Resident Pages: 0/65579  0/1G  0%
Elapsed: 0.001622 seconds

Running the test, then running vmtouch again

>> kvtest % ./kvtest run test1.db --count 100k --blob-api
SQLite version: 3.39.4
--count 100000 --max-id 100000 --asc
--cache-size 1000 --jmode delete
--mmap 0 --blob-api
Database page size: 4096
Total elapsed time: 2.140
Microseconds per BLOB read: 21.400
Content read rate: 467.2 MB/s


>> vmtouch test1.db                             
           Files: 1
     Directories: 0
  Resident Pages: 65579/65579  1G/1G  100%
         Elapsed: 0.012271 seconds

You can see that our entire database is cached now. And that’s why the second run is much faster.
If you run vmtouch test1.db again after some period of the first run you see a number smaller than 100%, which means the OS has evicted some pages already. So the longer you wait to execute the second run, the worse the performance will be.

>> vmtouch test1.db
           Files: 1
     Directories: 0
  Resident Pages: 51595/65579  806M/1G  78.7%
         Elapsed: 0.017729 seconds

Things you can try:

Before executing the first run, put all of test1.db into the cache by running vmtouch -t test1.db, then execute the test. You’ll already see good performance results in the first run;
You can execute the first run, evict all pages from the cache and execute again.

This confirmed that the performance is really coming from the OS disk page cache. And with that came, the question: isn’t the OS caching the files in the FileSystem approach? If not, why?

Playing directly with the files

Now I’ve tried to understand what was happening with the cache when running the test directly on the FileSystem.

>> ./kvtest run test1.dir --count 100k --blob-api
--count 100000 --max-id 1000 --asc
Total elapsed time: 0.776
Microseconds per BLOB read: 7.760
Content read rate: 1282.1 MB/s


>> kvtest % vmtouch test1.dir                             
           Files: 100000
     Directories: 1
  Resident Pages: 1000/100000  15M/1G  1%
         Elapsed: 1.0113 seconds

Only 1% of the files were cached, after the first run. That is pretty interesting. It seems that the caching behavior of the OS is a lot different when accessing the blobs via SQLite than via FileSystem. What if I force the files to be in the cache? You can do that by using the flags -dl. You can see the percentages increase until it reaches 100% and it stays that way.

>> vmtouch -dl test1.dir
>> vmtouch test1.dir                             
           Files: 100000
     Directories: 1
  Resident Pages: 50115/100000  783M/1G  50.1%
         Elapsed: 1.159 seconds


>> vmtouch test1.dir
           Files: 100000
     Directories: 1
  Resident Pages: 78288/100000  1G/1G  78.3%
         Elapsed: 1.0382 seconds


>> vmtouch test1.dir
           Files: 100000
     Directories: 1
  Resident Pages: 95351/100000  1G/1G  95.4%
         Elapsed: 1.0803 seconds


>> vmtouch test1.dir
           Files: 100000
     Directories: 1
  Resident Pages: 100000/100000  1G/1G  100%
         Elapsed: 1.0232 seconds

Running the test again, we get similar performance results:

>> ./kvtest run test1.dir --count 100k --blob-api
--count 100000 --max-id 1000 --asc
Total elapsed time: 0.736
Microseconds per BLOB read: 7.360
Content read rate: 1351.7 MB/s

So not only does the caching behave differently in the FileSystem approach, but it also does not impact the performance.

Final thoughts

I’m not an OS expert and have no idea how the cache replacement policy of an OS works but seems reasonable to expect 100,000 syscalls to open and read from a file would have an impact on the cache behavior; and also how doing so many syscalls seems to make the caching insignificant. In the SQLite approach, you are doing that only once. The article is pretty good but the way the benchmark was set up may lead to wrong conclusions if you’re not alert (not saying the article concluded things wrongly). Most of the performance comes from how the data was organized and accessed, which is usually a design decision a Software Engineer should make when targeting a use case.

Other than that, the greatest reward for reading the article was discovering vmtouch. Such a simple tool that enables awesome performance debugging. And also, I’m also in awe of how following your curiosity leads you to find things you could never have imagined before you started. One of the reasons I did not want to do a shallow reading on this article.

Making a change to SQLite source code

2022-10-18T00:00:00+00:00

Introduction

The other day, I was thinking about how I could get the bytes of a record of a recently inserted or updated row in SQLite. The motivation for that is that I wanted to create a hash of that row, essentially, to be able to build a Merkle Tree of the corresponding table as rows get inserted or updated.

The closest API that SQLite offers to what I was looking for is the sqlite3_update_hook:

The sqlite3_update_hook() interface registers a callback function with the database connection identified by the first argument to be invoked whenever a row is updated, inserted or deleted in a rowid table.

The issue with that API is that it only returns the rowid of the row. That means I would have to fetch all the columns for the rows inside the table. And even with that approach, I would still not get the raw bytes of the row record. I would just get the driver’s representation of that row.

There are probably plenty of approaches to how I could build such a tree, but as far as I’m concerned, the SQLite API does not offer what I was exactly looking for. So, I decided to dig deeper into the source code to see if there could be a world where I would understand it, and not only that, I could actually make some changes to it to provide what I was hoping for.

I have always felt intimidated by reading C language code. So I thought it would be just one of those times of opening a C language file and giving up minutes later. But it turns out this time was different.

Navigating the SQLite source code

I’ve cloned SQLitesource code using fossil and started navigating the files structure. Figure 1 - SQLite src directory If you’re somewhat familiar with how databases work you can probably imagine what some files are responsible for. Not bad. I decided to jump straight to the insert.c file to see if I could find something interesting there.

If you skim through all function implementation you’ll probably hit sqlite3Insert. Above the function signature, we see:

** This routine is called to handle SQL of the following forms:
**
**    insert into TABLE (IDLIST) values(EXPRLIST),(EXPRLIST),...
**    insert into TABLE (IDLIST) select
**    insert into TABLE (IDLIST) default values
**

Maybe inside this function, there was something that I could tweak. I was able to make some guesses about what’s happening in there, but what caught my eye was the number of function calls to functions of a name similar to sqlite3vdbeXXX.

That reminded me that SQLite uses a virtual machine called vdbe. That means all SQL statements are translated to the language of this virtual machine first. Then, the execution engine executes the virtual machine code. Let’s look at an example of how a simpler INSERT is translated into bytecode:

sqlite> create table a (a int, b text);
sqlite> explain INSERT INTO a VALUES (1, 'Hello');
addr  opcode         p1    p2    p3    p4             p5  comment      
----  -------------  ----  ----  ----  -------------  --  -------------
   Init           0     8     0                    0   Start at 8
   OpenWrite      0     2     0     2              0   root=2 iDb=0; a
   Integer        1     2     0                    0   r[2]=1
   String8        0     3     0     Hello          0   r[3]='Hello'
   NewRowid       0     1     0                    0   r[1]=rowid
   MakeRecord     2     2     4     DB             0   r[4]=mkrec(r[2..3])
   Insert         0     4     1     a              57  intkey=r[1] data=r[4]
   Halt           0     0     0                    0   
   Transaction    0     1     1     0              1   usesStmtJournal=0
   Goto           0     1     0                    0   

I concluded that all sqlite3Insert is really doing is translating a parsed INSERT statement to a bunch of virtual machine operations, according to all rules of SQLite insertion.

This was not really the place I was looking for. What I really needed was the place where the record is created before insertion. That could only be the place that is executing the virtual machine code, probably the place that is executing the Insert (OP_INSERT) opcode, I thought.

Looking at Figure 1, the vdbe.c file felt like a reasonable place to look for that. I went straight there.

What I found there was an 8000 lines switch( pOp->opcode ) statement and with a simple CMD+FOP_INSERT I found the place that was handling the Insert operation execution.

The first line of the case I found a hint of what I was looking for:

Mem *pData;       /* MEM cell holding data for the record to be inserted */

So pData points to the record data to be inserted. And you can see at [L54](https://github.com/sqlite/sqlite/blob/version-3.39.4/src/vdbe.c#L5402)02, pData = &aMem[pOp->p2];, how it is setting pData value to the virtual machine memory aMem address at the position pointed by virtual machine register p2.

Quick recap: at insert.c we’ve learned that the INSERT statement was translated into a bunch of virtual machine code. The data from the INSERT went to the virtual machine through those sqlite3vdbeXXX calls. I assume the call that registered the OP_INSERT opcode and the data into the virtual machine is the one at line 2593:

sqlite3VdbeAddOp3(v, OP_Insert, iDataCur, aRegIdx[i], regNewData);

And here is a nice description of regNewData :

** The regNewData parameter is the first register in a range that contains
** the data to be inserted or the data after the update.  There will be
** pTab->nCol+1 registers in this range.  The first register (the one
** that regNewData points to) will contain the new rowid, or NULL in the
** case of a WITHOUT ROWID table.  The second register in the range will
** contain the content of the first table column.  The third register will
** contain the content of the second table column.  And so forth.
**
** The regOldData parameter is similar to regNewData except that it contains
** the data prior to an UPDATE rather than afterwards.  regOldData is zero
** for an INSERT.  This routine can distinguish between UPDATE and INSERT by
** checking regOldData for zero.

So, at this point we are executing that machine code with that data. Scrolling a bit down, let’s see how pData is used. At L5448-L5449, we see:

x.pData = pData->z;
x.nData = pData->n;

And xis:

BtreePayload x;   /* Payload to be inserted */

Awesome. Scrolling a bit more, we see:

rc = sqlite3BtreeInsert(pC->uc.pCursor, &x,
    (pOp->p5 & (OPFLAG_APPEND|OPFLAG_SAVEPOSITION|OPFLAG_PREFORMAT)), 
    seekResult
);

Cool, we found the place where the raw record byte is being inserted. But, how do we know that it is the record formatted as documented here? If you look closely at the virtual machine code from our example INSERT, before the Insert opcode there is a MakeRecord opcode, that is responsible for building the record.

You can check the OP_MakeRecord implementation at vdbe.c file and see the following comment:

Convert P2 registers beginning with P1 into the [record format] use as a data record in a database table or as a key in an index.

And the last lines of the case statement, we find this:

/* Invoke the update-hook if required. */
if( rc ) goto abort_due_to_error;
if( pTab ){
  assert( db->xUpdateCallback!=0 );
  assert( pTab->aCol!=0 );
  db->xUpdateCallback(db->pUpdateArg,
         (pOp->p5 & OPFLAG_ISUPDATE) ? SQLITE_UPDATE : SQLITE_INSERT,
         zDb, pTab->zName, x.nKey);
}
break;

Yes! The update hook!

So, here I have everything I need in my hands. The update hook and the record bytes. So, I just need to update the callback to add the record, and voila.

Making changes to SQLite

That’s exactly what I did:

db->xUpdateCallback(db->pUpdateArg,
	(pOp->p5 & OPFLAG_ISUPDATE) ? SQLITE_UPDATE : SQLITE_INSERT,
	zDb, pTab->zName, x.nKey, pData->z, pData->n);

Passing the payload (pData->z) and its size (pData->n).

And, of course a bunch of more changes in multiple places to account for function signature change were necessary.

Here’s fossil status after the changes:

EDITED     src/main.c
EDITED     src/sqlite.h.in
EDITED     src/sqlite3ext.h
EDITED     src/sqliteInt.h
EDITED     src/tclsqlite.c
EDITED     src/vdbe.c

And the diff of changes in case you’re following along. I’ve compiled my changes following the instructions.

Forking Go’s SQLite driver

All right! Now it’s time to create a simple test of my change in a Go program. I’m most familiar with mattn/go-sqlite3 driver for interacting with SQLite. This project offers SQLite API access through Go by importing the SQLiteamalgamation file and working through C bindings.

So I forked that repo, and imported my own compiled amalgamation file. And did the necessary updates in the Go API to have access to that new value. My linter messed up with the diff. But only a few changes were needed:

A change at the updateHookTrampoline that now receives the record as *C.char and its size as int, and casts it to []byte passing it to callback:

func updateHookTrampoline(handle unsafe.Pointer, op int, db *C.char, table *C.char, rowid int64, data *C.char, size int) {
	callback := lookupHandle(handle).(func(int, string, string, int64, []byte))
	callback(op, C.GoString(db), C.GoString(table), rowid, C.GoBytes(unsafe.Pointer(data), C.int(size)))
}

And a change to RegisterUpdateHook.

Putting everything together

So, now we have everything in place to test this out. Let’s run a simple example inspired by SQLite Internals: Pages & B-trees blog post.

package main

import (
	"database/sql"
	"fmt"
	"log"
	"os"

	"github.com/mattn/go-sqlite3"
)

func main() {
	sqlite3conn := []*sqlite3.SQLiteConn{}
	sql.Register("sqlite3_with_hook_example",
		&sqlite3.SQLiteDriver{
			ConnectHook: func(conn *sqlite3.SQLiteConn) error {
				sqlite3conn = append(sqlite3conn, conn)
				conn.RegisterUpdateHook(func(op int, db string, table string, rowid int64, data []byte) {
					switch op {
					case sqlite3.SQLITE_INSERT:
						fmt.Printf("%x\n", data)
					}
				})
				return nil
			},
		})
	os.Remove("./foo.db")

	srcDb, err := sql.Open("sqlite3_with_hook_example", "./foo.db")
	if err != nil {
		log.Fatal(err)
	}
	defer srcDb.Close()
	srcDb.Ping()

	_, err = srcDb.Exec(`CREATE TABLE sandwiches (
		id INTEGER PRIMARY KEY,
		name TEXT,
		length REAL,
		count INTEGER
	);`)
	if err != nil {
		log.Fatal(err)
	}
	_, err = srcDb.Exec("INSERT INTO sandwiches (name, length, count) VALUES ('Italian', 7.5, 2);")
	if err != nil {
		log.Fatal(err)
	}
}

Don’t forget to add replace github.com/mattn/go-sqlite3 => github.com/brunocalza/go-sqlite3 v0.0.0-20220926005737-36475033d841 to go.mod .

If you run that, you’ll get:

05001b07014974616c69616e401e00000000000002

Which is exactly the raw record of the row ('Italian', 7.5, 2) as described at Efficient Sandwich Encoding, without the primary key and the length of the record (two first bytes).

And here I finish the journey. It was really fun to know I was able to understand some parts of SQLite source code, although I did not understand most of it, and make some changes and see those changes reflected through Go’s driver.

To be honest this approach of changing the source code of a database is too risky. Keeping up-to-date with new versions (and also I’d have to also keep an up-to-date fork of the Go’s driver) is too problematic.

I work on a web3 protocol called Tableland. Problems similar to this are always frequent in our Engineering team. If you enjoy this kind of stuff, get in touch through our Github or Discord, or @brunocalza.

There Are Many Ways To Safely Count

2021-07-08T00:00:00+00:00

Another day I was looking at a simple classic implementation of a shared counter in C++ using mutex, and I wondered what other thread-safe implementations existed. I usually use Go to explore my curiosity. The result of this exploration is a compilation of ways on how to implement a goroutine-safe counter.

Don’t Do This

Let’s start with the non-safe implementation.

type NotSafeCounter struct {
    number uint64
}

func NewNotSafeCounter() Counter {
    return &NotSafeCounter{0}
}

func (c *NotSafeCounter) Add(num uint64) {
    c.number = c.number + num
}

func (c *NotSafeCounter) Read() uint64 {
    return c.number
}

Nothing magical. Let’s test its correctness by running 100 *goroutines* where 2 thirds of them Add 1 to the shared counter.

func testCorrectness(t *testing.T, counter Counter) {
    wg := &sync.WaitGroup{}
    for i := 0; i < 100; i++ {
        wg.Add(1)
        if i%3 == 0 {
            go func(counter Counter) {
                counter.Read()
                wg.Done()
            }(counter)
        } else if i%3 == 1 {
            go func(counter Counter) {
                counter.Add(1)
                counter.Read()
                wg.Done()
            }(counter)
        } else {
            go func(counter Counter) {
                counter.Add(1)
                wg.Done()
            }(counter)
        }
    }

    wg.Wait()

    if counter.Read() != 66 {
        t.Errorf("counter should be %d and was %d", 66, counter.Read())
    }
}

The result of the test is not deterministic. Sometimes it passes. But sometimes you get a message like counter_test.go:34: counter should be 66 and was 65.

The Classic

The traditional way to implement a correct counter is to use a mutex that guarantees that only one operation is done at a time. In Go, we simply use the sync package.

type MutexCounter struct {
    mu     *sync.RWMutex
    number uint64
}

func NewMutexCounter() Counter {
    return &MutexCounter{&sync.RWMutex{}, 0}
}

func (c *MutexCounter) Add(num uint64) {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.number = c.number + num
}

func (c *MutexCounter) Read() uint64 {
    c.mu.RLock()
    defer c.mu.RUnlock()
    return c.number
}

Now the tests run deterministically and always pass.

Using Channels

Locks are low-level primitives that let you achieve synchronization. Go offers a more high-level primitive called channels. There are a lot mutexes-versus-channels, which-one-is-better or which-one-should-I-use kinds of discussions about mutexes and channels. Some of the discussions are valid and very interesting, but that is not the point of this blog post.

The way we are going to implement a goroutine-safe counter using channels is by having a channel where every operation (Add or Read) called on the counter will be queued in a channel. The operations will be represented as a function func(). When created, the counter spawns a goroutine that executes the queued operations in serial order.

Here is the counter definition:

type ChannelCounter struct {
    ch     chan func()
    number uint64
}

func NewChannelCounter() Counter {
    counter := &ChannelCounter{make(chan func(), 100), 0}
    go func(counter *ChannelCounter) {
        for f := range counter.ch {
            f()
        }
    }(counter)
    return counter
}

See how the counter’s goroutine only reads the operations from the channel and executes them.

When a goroutine calls Add, we queue a write operation:

func (c *ChannelCounter) Add(num uint64) {
    c.ch <- func() {
        c.number = c.number + num
    }
}

When a goroutine calls Read, we queue a read operation:

func (c *ChannelCounter) Read() uint64 {
    ret := make(chan uint64)
    c.ch <- func() {
        ret <- c.number
        close(ret)
    }
    return <-ret
}

What I really like about this implementation is how clear it is to visualize the operations being executed in serial order.

The Atomic Way

We can use even lower-level primitives and execute atomic instructions provided by the sync/atomic package.

type AtomicCounter struct {
    number uint64
}

func NewAtomicCounter() Counter {
    return &AtomicCounter{0}
}

func (c *AtomicCounter) Add(num uint64) {
    atomic.AddUint64(&c.number, num)
}

func (c *AtomicCounter) Read() uint64 {
    return atomic.LoadUint64(&c.number)
}

Compare And Swap

Alternatively, we can use the very classical atomic primitive Compare And Swap to Add a number to the counter.

func (c *CASCounter) Add(num uint64) {
    for {
        v := atomic.LoadUint64(&c.number)
        if atomic.CompareAndSwapUint64(&c.number, v, v+num) {
            return
        }
    }
}

func (c *CASCounter) Read() uint64 {
    return atomic.LoadUint64(&c.number)
}

Basically, it tries infinitely until it successfully updates the counter correctly.

What About Float Types?

In my exploration, I came across an awesome talk, called Prometheus: Designing and Implementing a Modern Monitoring Solution in Go, that discusses these techniques and benchmarks them. At the final, it talks about how to implement a counter of floats. All techniques provided so far works for floats, except the ones that use sync/atomic. sync/atomic does not provide atomic operations on floats. In the video, Björn Rabenstein presents how to solve this by storing the float as an uint64 and use the math.Float64bits and math.Float64frombits to do the conversion between float64 and uint64.

type CASFloatCounter struct {
    number uint64
}

func NewCASFloatCounter() *CASFloatCounter {
    return &CASFloatCounter{0}
}

func (c *CASFloatCounter) Add(num float64) {
    for {
        v := atomic.LoadUint64(&c.number)
        newValue := math.Float64bits(math.Float64frombits(v) + num)
        if atomic.CompareAndSwapUint64(&c.number, v, newValue) {
            return
        }
    }
}

func (c *CASFloatCounter) Read() float64 {
    return math.Float64frombits(atomic.LoadUint64(&c.number))
}

Final Words

This is a simple collection of implementations of a shared counter. It is the result of my curiosity and also the result of trying to achieve a fundamental understanding of concurrency. If you know more ways on how to do this, I’d love to know.

You can check the implementations, run the tests and benchmarks at brunocalza/sharedcounter.

I’m always trying to share what I am learning about database internals.

I’m on Twitter.

Getting To Know Logical Clocks By Implementing Them

2021-07-02T00:00:00+00:00

The Clock Synchronization and Ordering Problems

A single node system has no problem deciding what time it is and which order the events inside the system happened. The node has a timer, called clock, and any process that needs to make use of time makes a call to the operating system. If process (a) makes use of time before a second process (b), the time read by (a) will be smaller than the time read by (b). In a more formal way,

\[a \to b \implies T(a) < T(b)\]

This is a very natural condition. If something happens before another thing, it is expected that the time at the first thing occurred to be smaller. The operator ( $\to$ ) is called happened-before, and it is defined as:

If (a) and (b) are events in the same process, and (a) occurs before (b), then (a $\to$ b)
If event (a) is the sending of a message from a process, and event (b) is the receiving of the same message by another process, then (a $\to$ b )
If (a $\to$ b) and (b $\to$ c), then (a $\to$ c)

In a distributed system, however, the above condition is not so easy to maintain. We have multiple nodes. Each with its own clock. A clock is an electronic circuit present in the hardware made of a crystal that oscillates at a constant frequency. We call physical clocks this kind of clock that measures real-time. The problem in a distributed system is that there is no guarantee that the rate at which each one of the clocks oscillates will be the same, a phenomenon known as clock drift. This is the clock synchronization problem. Nodes can’t agree on which time it is. Of course, measuring the real-time is important to many applications, so solutions to coordinate the clocks as NTP have been in use. However, even when using something like NTP and node’s physical clock times becoming very close within a bounded range, it is still possible that an event happening before another to have a superior timestamp. There are many applications that it is important to have an agreement on the order of events. These applications cannot rely on physical clocks.

Lamport Clocks

In 1978, Leslie Lamport tackled this problem in the paper Time, Clocks, and the Ordering of Events in a Distributed System and presented a logical clock implementation built in a way to satisfy the above condition and it is able to make nodes agree on an order in which events occur.

A logical clock is just a counter that holds a number, called timestamp. This number has no relationship to physical time. Its only purpose is to capture the ordering of events. And in order to have this property, the clock needs to satisfy $a \to b \implies T(a) < T(b) $ for any events $a$ and $b$. Lamport calls it Clock Condition. In other words, each node of a distributed system will have a clock (logical), that is just a counter; and all of the counters store numbers that are in agreement that if something happened the timestamp of this happening will be smaller than the next event.

It is not hard to come up with a counter that captures the condition. Let’s break things into two parts: events happening inside a process and events happening between processes (messages exchange). If two events happen locally, a counter that increments its value every time an event satisfies the Clock Condition. For the second part, we have to satisfy the fact $C_i(a) < C_j(b)$, where $a$ is the event of sending a message at process $i$ and event $b$ is the receiving the same message at process $j$; $C_x$ is the value of the clock at process $x$. Lamport proposes the sending of the timestamp together with the message and the receiver to update its counter to the greater value between its timestamp and the received one, upon receiving the message.

The Algorithm

Each node in a distributed system has a clock, which is a counter that stores the time t that the last event occurred. The counter starts at t = 0.

When an event occurs at the node, t is incremented, that is, t = t + 1.

When a node wants to send a message m to another node, t is incremented and it is sent together with the message, t = t + 1, then, send(m, t).

When a node receives a message, it updates its own clock with the maximum of its current time t and the time t_received that came with the message and adds one. That is, t = max(t, t_received) + 1.

Visualizing

I built a simple tool in Go that helps with defining the events in concurrent goroutines and visualizing the message flow and clock values. It is highly based on an implementation done by Michael Whittaker. You can plug your own example (like example3) and execute it by running go run cmd/* Example3 | ./plot.py . Execution of Example 3 Every event has a mark on the timeline. A line in the figure is the connection between two events and it increases from the sender to the receiver.

It is very clear from the figure that the Clock Condition holds. Any pair of events that has a happened-before relation can be compared. For example, $ (1, A) \to (3, A)$; or $ (1, C) \to (8, B)$.

Does $ (1, A) \to (6, C)$? Yes. According to the definition, $ (1, A) $ happened-before $ (6, C) $ by transitivity. There is a sequence of events from $ (1, A)$ to $ (6, C)$: $ (1, A), (3, B)$, $(4, B)$, $(5, B)$ and $(6, C)$. Be aware that for some pair of events we can’t establish a happened-before relation according to our definition, for example, the pair $(3, A)$ and $(4, B)$. We say that these events are concurrent.

Total ordering

The usefulness of Lamport Clocks comes from the fact that it can be used to define a total order relation among all events in the system. In Lamport words,

We can use a system of clocks satisfying the Clock Condition to place a total ordering on the set of all system events. We simply order the events by the times at which they occur. To break ties, we use any arbitrary total ordering of the processes.

Being able to totally order the events can be very useful in implementing a distributed system. In fact, the reason for implementing a correct system of logical clocks is to obtain such a total ordering.

The use of Lamport Clocks and total ordering in a distributed system is what let us coordinate or synchronize events of multiple nodes in a way that every node agrees. It is a simple consensus algorithm.

Limitations

One limitation of Lamport Clocks is that the inverse of the Clock Condition does not hold:

\[T(a) < T(b) \implies a \to b\]

So it can’t be used to determine if one event happens before the other given both timestamps. And also, it can’t help us determine if two events are concurrent or not by comparing the timestamps only.

These limitations are overcome by Vector Clocks, the second kind of logical clock.

Vector Clocks

Vector Clocks is an improvement of Lamport Clocks, because it not only captures the happened-before ($ \to $) relationship, that physical clocks do not have, but it also has the property of telling if events are concurrent or not, a property that Lamport Clocks don’t have. Instead of holding only the timestamp of its own process, each clock holds a vector of timestamps, containing the timestamps of all processes. Apparently, it was first mentioned in 1986 at a paper called Highly-Available Distributed Services and Fault-Tolerant Distributed Garbage Collection.

The Algorithm

Each node stores a vector of timestamps $t = \langle t_1, t_2, \ldots, t_n \rangle $ of size n, where n is the number of nodes in the distributed systems. It is initialized at $ \langle 0, 0, \ldots, 0 \rangle $.

When an event occurs at a node i, it increments the ith entry of the t vector t[i] = t[i] + 1.

When a node i sends a message to another node, it increments the ith entry of the t vector t[i] = t[i] + 1 and attaches the vector with the message.

When a node i receives a message it updates its vector t using an element-wise maximum operation between its vector and the received vector t_received. Then, increments the ith entry of the t vector t[i] = t[i] + 1 .

Properties

Let’s use the same tool to visualize the same sequence of events of the example before (this example can be found at example8.go). The first thing to observe is that we don’t need to add the label of the process to uniquely identify a timestamp. A vector timestamp uniquely identifies an event. Second, we can see that the vector clock of an event is a counter of the number of events that have a happened-before relationship to the event. For example, the event $($(3,1,0)$)$ informs that 2 events at A and 1 event at B happened-before it. And that is true, since the events $($(1,0,0)$)$, $($(0,1,0)$)$ and $($(2,1,0)$)$ happened-before $($(3,1,0)$)$. Another way of putting it: a vector timestamp of an event $e$ represents a set of events, $e$ and its causal dependencies: $ {e} \cup {a \mid a \to e} $.

Vector Clocks improves on Lamport Clocks on the fact that now you can compare event timestamps and decide which one happened before the other or if there are concurrent. We already said a vector timestamp uniquely identify an event, that is $ V(a) = V(b) \iff a = b $ for events $a$ and $b$. We can also compare if a vector timestamp is less ($<$) than another: for every element of the first vector it should be less or equal to the corresponding element of the second vector and the vectors can’t be the same. The interesting fact is that $ V(a) < V(b) \iff a \to b $ for events $a$ and $b$. This means that given two vector clocks we can decide if one happened-before the other. If the events are not comparable by the equals ($=$) operator or the less ($<$) operator, we say that the events are concurrent. For example, we know that $(1,0, 0) \to (1, 5, 4)$ because $(1,0, 0) < (1, 5, 4)$. And we know that $(4,6,0)$ and $(0,0,1)$ are concurrent because they are not comparable.

Wrapping Up

This was a discussion on two popular kinds of logical clocks, their properties, and their implementations. Logical Clocks (not exactly the implementations provided above) are very important in distributed systems and very popular among databases for detecting conflicts of data versioning. Take a look at an excerpt from the Dynamo paper:

One can determine whether two versions of an object are on parallel branches or have a causal ordering, by examining their vector clocks. If the counters on the first object’s clock are less-than-or-equal to all of the nodes in the second clock, then the first is an ancestor of the second and can be forgotten. Otherwise, the two changes are considered to be in conflict and require reconciliation.

We can find examples on Riak KV that uses logical clocks to track the history of updates to values, and detect conflicting writes, and also on CockroachDB relies on hybrid logical clocks to provide serializability.

I hope you enjoyed the content. Consider following me on Twitter if that’s the case. I write about databases internals while I learn more about it.

References

What Zero-copy Serialization Means?

2021-06-01T00:00:00+00:00

I was reading about serialization formats the other day and came across the last column “Supports Zero-copy operations”. I had no idea of what it meant. Moments before I got on this Wikipedia page, I was looking into how to serialize a struct in *Go, *without using any specific format, just raw serialization (don’t even know if the term raw serialization means anything). While searching for a way I stumbled upon this Stack Overflow answer:

… if you consent to unsafety and actually need to read struct as bytes, then relying on byte array memory representation might be a bit better than relying on byte slice internal structure.
type Struct struct {
   Src int32
   Dst int32
   SrcPort uint16
   DstPort uint16
}

const sz = int(unsafe.SizeOf(Struct{}))
var asByteSlice []byte = (*(*[sz]byte)(unsafe.Pointer(&struct_value)))[:]
Works and provides read-write view into struct, zero-copy.

I became intrigued by the term and decided to research a little bit. The first results were about the zero-copy strategy of copying data minimizing context switches between the kernel space and user space. I had no clue this existed and I was very excited to know about this. Learned about the sendfile system call and found an awesome blog post about it by @b0rk. But it seemed to me that this had nothing to do with the zero-copy meant in the serialization context. So I favorited a bunch of web pages about the operational system zero-copy and resumed my research about serialization zero-copy.

For some reason, I could only find discussions about zero-copy tied to the Cap’n Proto serialization protocol. Cap’n Proto is a serialization protocol (after v0.4 it also became a RPC protocol) created Kenton Varda, which worked on Protocol Buffers version 2. According to him, Cap’n Proto is the result of years of experience working on Protobufs, listening to user feedback, and thinking about how things could be done better.

The marvelous of Cap’n Proto is that it has no cost of serialization/deserialization. This is because of its zero-copy implementation. But what zero-copy means? The first good clarification for me came when reading the comparison of zero-copy protocols to Protocol Buffer at Cap’n Proto, FlatBuffers, and SBE:

Zero-copy

The central thesis of all three competitors is that data should be structured the same way in-memory and on the wire, thus avoiding costly encode/decode steps.

Protobufs represents the old way of thinking.

On the Cap’n Proto home page, it says:

Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk!

Things became a lot clearer to me. It is as if the “serialization” implicitly happened when the object was built, and you already have the bytes in your hands. I found an interesting question on the Cap’n Proto forum: What does zero-copy mean?. The discussion revolves around the fact that zero-copy protocols are not well suitable for an object that mutates state as pointed in Cap’n Proto, FlatBuffers, and SBE:

Usable as mutable state

Protobuf generated classes have often been (ab)used as a convenient way to store an application’s mutable internal state. There’s mostly no problem with modifying a message gradually over time and then serializing it when needed.

This usage pattern does not work well with any zero-copy serialization format because these formats must use arena-style allocation to make sure the message is built in contiguous memory. Arena allocation has the property that you cannot free any object unless you free the entire arena. Therefore, when objects are discarded, the memory ends up leaked until the message as a whole is destroyed. A long-lived message that is modified many times will thus leak memory.

So, if it not recommended to mutate the state of the Cap’n Proto message, doesn’t that imply that a copy is needed to perform the mutation in another structure? You can check Kenton’s answer in the thread.

I have also come across an interesting and intense discussion on Hacker News about the definition of zero-copy. Most of the discussion was around the usefulness and the trade-offs of the definition. I am just going to highlight Kenton’s definition:

Some people use the term “zero-copy” to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message buffer, rather than having to allocate a copy of the bytes at parse time.

Cap’n Proto and FlatBuffers implement a much stronger form of zero-copy. With them, it’s not just strings and byte buffers that are zero-copy, it’s the entire data structure. With these systems, once you have the bytes of a message mapped into memory, you do not need to do any “parse” step at all before you start using the message.

All this made things much clearer to me. But I still wondered about the claim made on that Stack Overflow post. Isn’t that assignment a copy of data? And the answer to that I found on Go Slices: usage and internals

Slicing does not copy the slice’s data. It creates a new slice value that points to the original array. This makes slice operations as efficient as manipulating array indices.

Stumbled upon a lot of new things that I will surely explore further and learned a lot from this quest. I was recently working on an implementation of database storage using the concept of slotted paged. I was basically doing raw manipulation of bytes and pointers in order to insert a tuple into a table. I wonder if one can make use of a protocol such as Cap’n Proto to make database storage more reliable and faster. That’s it for this post.

The Cache is Full

2021-05-23T00:00:00+00:00

In computing, caching is all over the place. It is found in hardware (CPU and GPU), operating system’s virtual memory, buffer pool managers inside databases, in the Web (Content Delivery Networks, browsers, DNS servers …), and also inside the applications we build. In this post, we take an abstract look of what cache is, the replacement problem that arises from cache’s own nature and some algorithms for implementing cache replacement policies.

Defining Cache

Independently of the context of where caching is being applied, the pure reason for its existence is to a system have better performance than it would without the cache. A system works “fine” without caching. It is just that by applying this technique the system performs better. And, in the context of caching, perform better means accessing the data that is needed faster.

In order to achieve the performance gain, part of the data is replicated to a secondary storage. Instead of accessing the data through the primary storage, the system accesses through the cache; and the access through the cache is supposed to be more performative. The performance gain of accessing this different storage may arise for multiples reasons. It can be for a technological one. For example, the data in the cache is in memory and the primary storage uses a disk. It can be to avoid overheads. For example, the primary storage can be a relational database, and you add a cache to avoid the overhead of connecting to the database, parsing the SQL query, accessing the disk, and transforming the results to the format that suits the client application. It can be for geographical reasons, as in the case of Content Delivery Networks. Cache as a copy of a subset of the primary storage One of the main drivers of why the cache contains only a subset of the primary storage is cost. Maintaining a copy of the entire primary storage may be too costly, so we have to content ourselves with a cache of limited capacity. More importantly, by design, a cache is supposed to have small sizes. That’s because the whole purpose of caching is to optimize for data access. Most of the data is not accessed, so keeping in cache a subset of data that is frequently accessed is what allows faster look-ups. With all this in mind, let’s define cache, for the purposes of this post, as:

*A secondary storage with limited capacity that contains a copy of a subset of the data that is stored in main storage. *

By copy here, I don’t mean exactly copy. Data stored in cache can be the result of computation like the database example talked about above.

The “Cache is Full” Problem

What would be the optimal performance of a cache design? Every time that data needs to be accessed, it can be found in the cache (cache hit). But since the cache is only a subset, it means that sometimes data won’t be found in cache (cache miss). If the data that is not in the cache is never requested, it doesn’t really matter if it is not in the cache. So, as cache designers, we have to make a decision of what data should we keep in the cache and what data should we keep out of the cache in order to maximize the cache hit (or minimize cache miss).

To make things clear and for the sake of the discussion let’s assume the following while interacting with the cache:

The cache starts empty;
The application always interacts with the cache first. If it doesn’t find the data that is looking for (Get operation), it consults the primary storage, then it adds to the cache (Put operation), as in the figure below;
The entries inside the cache never expires (there is no TTL);
The entries in the cache are always up-to-date.

On a cache miss, the primary storage has to be consulted. Decision on what data lies in cache has to be made There are multiple architectural options on how to interact with the cache. That is not the point to be discussed here. The point is what data should we keep in cache. From the above figure, there are two points where we have to make a decision of what data lies in the cache. The first point in the Put operation. The client doesn’t need to add every missed entry to the cache, it can wisely choose those that make more sense. The second decision point is when the client decides to add an entry to the cache but the cache is full. Full may mean that the numbers of entries reached maximum capacity or that the cache has reached its maximum pre-defined storage or any other criteria that are more relevant to your scenario. When that happens, a decision on what existing entry needs to be evicted from cache to make room for a new entry has to be made. This decision-making process, which is the topic of interest in the post, is called cache replacement policy.

Cache Replacement Policies

The problem of choosing an entry to discard from cache can be re-framed as discarding the entry that is least likely to be used in the future. This is not an easy problem, because we can’t predict which entries are not going to be needed in the future. If we could predict, we’d have an optimal algorithm.

Cache replacement policies rely on heuristics that are dependent on the context. A policy for a CPU cache will probably differ from the policy of web caching. Hardware caching may impose architectural restrictions that may be not found on the web. Besides that, the parameters that are aimed for optimization influence a lot on the chosen algorithm. The most common parameter that is attempted to maximize is the hit ratio. Some other parameters that can be considered for optimization are byte hit ratio, cost of access, and latency.

In addition, these policies can make use of characteristics of the cache entry to base a decision. Some of the most popular characteristics are:

recency: time of (since) the last reference to the entry;
frequency: number of requests to an entry;
size: the size of the entry;
cost of fetching the object: cost to fetch an object from its origin;

Because of the context, the parameters that are being optimized, and the information used in the heuristics, an enormous variety of replacement algorithms arose. We are going to focus on FIFO, LRU, CLOCK, and LFU to get a full understanding of how a replacement policy typically works.

Implementing Some Policies in Go

The cache

We are going to make clear distinctions about the core logic of the cache and the logic of the replacement policy. Not only that, we are going to have multiples policies that implement the same policy interface. This way it makes it easier to swap implementations to see how the policies differ from one another. In practice, such decoupled implementation adds some memory overhead (you need an additional hash table) and it is quite hard to have an interface that abstracts all possible caching policies. In our case, the abstraction works because the chosen algorithms are somewhat related.

Our cache is going to be represented by

type CacheKey string

type CacheData map[CacheKey]string

type Cache struct {
    maxSize int
    size    int
    policy  CachePolicy
    data    CacheData
}

For simplicity will have both the key and the value as string. There is one attribute, size, for keeping track of the cache’s size and another indicating its capacity, maxSize. It has a policy, that will be responsible for evicting cache entries when the cache is full. Lastly, it uses a hash table for storing the cache entries.

A cache policy is defined as

type CachePolicy interface {
    Victim() CacheKey
    Add(CacheKey)
    Remove(CacheKey)
    Access(CacheKey)
}

Victim runs the policy algorithm and elects a CacheKey , called victim, for removal;
Add makes a cache key eligible for eviction
Remove makes a cache key no longer eligible for eviction
Access indicates to the cache policy that a cache key was accessed. This provides additional information to the cache replacement algorithm to make its decision

By decoupling the core logic of the cache and the replacement logic, it is very easy to understand how a cache works. Here is the Put operation

func (c *Cache) Put(key CacheKey, value string) {
    if c.size == c.maxSize {
        victimKey := c.policy.Victim()
        delete(c.data, victimKey)
        c.size--
    }
    c.policy.Add(key)
    c.data[key] = value
    c.size++
}

If the cache is full, it calls the replacement policy Victim that runs the algorithm and returns a cache key for eviction. The cache, then, removes the cache entry making room for the new key. The Get operation is straightforward. If it finds the key in the cache, it indicates to the key was accessed and returns the cached value.

func (c *Cache) Get(key CacheKey) (*string, error) {
    if value, ok := c.data[key]; ok {
        c.policy.Access(key)
        return &value, nil
    }

    return nil, errors.New("key not found")
}

We have a cache implementation. Now let’s focus on the replacement policies.

FIFO

FIFO (First In First Out) is one of the simplest algorithms. It simply removes the entries in the order that they were added. So, the decision that the algorithm does is: choose the oldest entry in the cache for removal.

It can be easily implemented using a queue. A O(1) queue is usually implemented using a doubly linked list . We are going to really on Go’s container/list implementations of a doubly-linked list.

Here is the FIFO policy definition

type FIFOPolicy struct {
    list    *list.List
    keyNode map[CacheKey]*list.Element
}

We are going to make use of an additional hash table, to make it possible to make the Remove method implementation O(1). If you add 5 elements to the cache, the FIFOqueue will look like this: The state of the FIFO queue Choosing a key for eviction is as simple as choosing the element that lies at the tail of the list. The methods’ implementations are just simple list operations.

func (p *FIFOPolicy) Victim() CacheKey {
    element := p.list.Back()
    p.list.Remove(element)
    delete(p.keyNode, element.Value.(CacheKey))
    return element.Value.(CacheKey)
}

func (p *FIFOPolicy) Add(key CacheKey) {
    node := p.list.PushFront(key)
    p.keyNode[key] = node
}

func (p *FIFOPolicy) Remove(key CacheKey) {
    node, ok := p.keyNode[key]
    if !ok {
        return
    }
    p.list.Remove(node)
    delete(p.keyNode, key)
}

One interesting thing is the implementation of the Access method, which is empty. Since the FIFO policy does not rely on access patterns to make its decision, the *Get operation *has no effect on the algorithm.

func (p *FIFOPolicy) Access(key CacheKey) {
}

LRU

LRU (Least Recently Used) is one of the most popular cache replacement policies. If an item is in the cache for a long time but was used recently, it will probably not be elected. The items that have not been used for a while are the ones considered for eviction. If you think about it, this algorithm is very similar to the FIFO algorithm. We just have to take into account the fact that an entry was used recently. In fact, this algorithm can be implemented identically as the FIFO, excepts for the Access method.

func (p *LRUPolicy) Access(key CacheKey) {
    p.Remove(key)
    p.Add(key)
}

The LRU relies on access patterns. Every time a cache entry is accessed, we update the recency of the cache key in the policy by removing the key from the queue and pushing it to the front, where the most recent item lies.

Let’s do some calls and how the internal LRUqueue behaves.

cache := NewCache(5, LRU)

cache.Put("1", "1")
cache.Put("2", "2")
cache.Put("3", "3")
cache.Put("4", "4")
cache.Put("5", "5")

cache.Get("3")
cache.Get("1")

How the queue changes after the Get operation If we call cache.Victim() now, it will choose 2 which is the least recent. 2 is evicted to make room for 6

CLOCK

In some situations, the LRU policy is not considered because accessing a cache entry involves removing the entry from the back of the list and reinserting it at the front. For an operating system, this operation is very costly. Since the LRU is generally considered a good policy, alternatives that approximate it are used instead. One of the alternatives is called CLOCK. Like the LRU, it is an improvement from FIFO. However, it avoids the costly operation of manipulating the list on every access making use of a circular list (hence the name CLOCK) a reference bit to indicate that the entry was recently accessed. Here is the representation of its data structure. Representation of the circular list The eviction algorithm makes use of a clock pointer to traverse the circular list in search of an entry that has the reference bit set to 0 (ref = 0). Every time that it checks the reference bit, if it is set to 1, it sets it to 0 and goes for the next entry. To illustrate this, let’s look at the state of the clock after adding entry 6 at the above clock. After Put 6 Using Redis as an LRU cache

It will traverse the circular list setting every reference bit to 0 until it reaches the first entry of reference bit 0, which will be entry 1. It will evict the 1, to make room for the 6. Now let’s do two Get operations: Get 2 and Get 3. They are going to set the reference bit to 1. After Get *operations When we look at the implementation of the Access method, it is clear that there is no need to manipulate the list as the *LRU did. It directly accesses the node and changes its reference bit.

func (p *ClockPolicy) Access(key CacheKey) {
    node, ok := p.keyNode[key]
    if !ok {
        return
    }
    node.Value = &ClockItem{key, true}
}

If we try to add a new entry 7 now, it will set the reference bits of 6, 2, and 3 to 0, and then evict entry 4. After Put 7 Here is how we represent the policy in code:

type ClockItem struct {
    key CacheKey
    bit bool
}

type ClockPolicy struct {
    list      *CircularList
    keyNode   map[CacheKey]*ring.Ring
    clockHand *ring.Ring
}

CircularList is an implementation that makes use of container/ring, Go’s implementation of a circular list.

And the Victim method described above is implemented with a for loop that advances the clock hand checking the references bit.

func (p *ClockPolicy) Victim() CacheKey {
    var victimKey CacheKey
    var nodeItem *ClockItem
    for {
        currentNode := (*p.clockHand)
        nodeItem = currentNode.Value.(*ClockItem)
        if nodeItem.bit {
            nodeItem.bit = false
            currentNode.Value = nodeItem
            p.clockHand = currentNode.Next()
        } else {
            victimKey = nodeItem.key
            p.list.Move(p.clockHand.Prev())
            p.clockHand = nil
            p.list.Remove(&currentNode)
            delete(p.keyNode, victimKey)
            return victimKey
        }
    }
}

You can check how the clock replacement can be used in a buffer pool manager at How Buffer Pool Works: An Implementation In Go.

LFU

The LFU (Least Frequently Used) cares more about frequency than recency. It relies on how many accesses an entry had, which means that somehow the algorithm needs to keep track of the frequency count. Historically, the most popular implementation uses a min-heap to keep track of the least frequently used entries. A heap is a good data structure choice, but it offers O(log n) operations. We are going to implement an algorithm with O(1) operations, by keeping a hash table the maps the frequency to a doubly-linked list. Each frequency has its own queue. Just because this implementation is O(1) it doesn’t mean it better. It has additional memory costs that may not be appropriate depending on the context.

Here is the LFU policy definition:

type Frequency int

type LFUItem struct {
    frequency Frequency
    key       CacheKey
}

type LFUPolicy struct {
    freqList     map[Frequency]*list.List
    keyNode      map[CacheKey]*list.Element
    minFrequency Frequency
}

We are going to store an LFUItem as the value of the list element. The LFUItem contains the frequency, which is helpful when updates to the frequency list need to be made. The minFrequency keeps track ofImplementations the minimum frequency that can be found. That is helpful to locate the list from which some element will be chosen to evict.

Here is a representation of how the algorithm works (from left to right) showing the state of the frequency list after some operation. Adding a new entry to the cache means pushing the cache key of that entry to the front of the queue of freqList[1]. Frames 1, 2, 4, 6, 7 show that. Here is the implementation.

func (p *LFUPolicy) Add(key CacheKey) {
    _, ok := p.freqList[1]
    if !ok {
        p.freqList[1] = list.New()
    }

    node := p.freqList[1].PushFront(LFUItem{1, key})
    p.keyNode[key] = node
    p.minFrequency = 1
}

When a Get operation occurs, it follow that we need to remove the element where it currently lies and push to the front of list stored at the freqList[frequency + 1].

func (p *LFUPolicy) Access(key CacheKey) {
    node := p.remove(key)

    frequency := node.Value.(LFUItem).frequency
    _, ok := p.freqList[frequency+1]
    if !ok {
        p.freqList[frequency+1] = list.New()
    }

    node = p.freqList[frequency+1].PushFront(LFUItem{frequency + 1, key})
    p.keyNode[key] = node
}

Lastly, the element the is chosen for eviction is the one that lies at the tail of the minFrequency list. In our example, would be entry 4. And here is how to implement it:

func (p *LFUPolicy) Victim() CacheKey {
    fList := p.freqList[p.minFrequency]
    element := fList.Back()
    fList.Remove(element)
    delete(p.keyNode, element.Value.(LFUItem).key)
    return element.Value.(LFUItem).key
}

Final Considerations

In this post, we tried to take an abstract look at the cache and what happens when the cache is full. We discussed and showed how to implement 4 popular cache replacement policies, that can be found at brunocalza/cache-replacement-policies. The aim of this post was not to discuss the trade-offs or in which circumstances one approach is better than another. That is a huge topic and full of nuances. Having said that if you know how these techniques are applied in real systems feel free to share. Consider following me on Twitter, if you enjoy the content.

Further Materials and References

Meizhen, Weng, Shang Yanlei, and Tian Yue. “The design and implementation of LRU-based web cache.” 2013 8th International Conference on Communications and Networking in China (CHINACOM). IEEE, 2013.
Podlipnig, Stefan, and Laszlo Böszörmenyi. “A survey of web cache replacement strategies.” ACM Computing Surveys (CSUR) 35.4 (2003): 374-398.
Balamash, Abdullah, and Marwan Krunz. “An overview of web caching replacement algorithms.” IEEE Communications Surveys & Tutorials 6.2 (2004): 44-56.
Wikipedia Cache (computing)
Wikipedia Page Replacement Algorithm
Clock Page Replacement
HashiCorp LRU’s library: hashicorp/golang-lru
Caching in theory and practice
Least Frequently Used (LFU) Cache Implementation
Constant Time LFU
When and Why to use a Least Frequently Used (LFU) cache with an implementation in Golang
Using Redis as an LRU cache

How Buffer Pool Works: An Implementation In Go

2021-02-11T00:00:00+00:00

I have been exploring how disk-oriented databases efficiently move data in and out of disk. One way, that I explored in Discovering and exploring mmap using Go and But how, exactly, databases use mmap?, is through memory-mapped files. Although mmap is a really neat solution, it has some troubles. Most troubles come from the fact that the database has no control of how pages are flushed to disk since that job is carried through the OS. Because of that most databases avoid using mmap. However, they still need to read and write data from disk in an efficient manner. And the answer to that is: buffer pool.

In this post, we’ll explain how a buffer pool manager works and how to implement one in Go. But first, let’s do a simple recap of how data is structured in disk.

How a database file is structured?

The file in disk that stores data of a database is just, in most databases, an array of bytes organized into fixed-length blocks called pages. A database file is a linear array of continuous pages. A database file is organized into fixed-length blocks called pages The page length is usually a multiple of the filesystem block length ranging from 512 bytes to 16KB. PostgreSQL, for example, uses 8KB, although a different page size can be selected when compiling the server¹. InnoDB, a storage engine used in MySQL, uses 16KB for page size².

The layout of these pages is not the scope of this post. However, just to give a glimpse of what happens when writes are executed, think that, for row-oriented databases, tuples are appended inside the page until it is filled up. When the page is filled up, another page needs to be allocated to support incoming writes.

The buffer pool manager

So we have our data in disk and the database wants to access it, let’s say, through a SELECT command. Or we have new incoming data, that we want to store. In order to manage reads and writes efficiently, the common practice is to avoid touching the disk directly. The execution engine that is processing the commands has no access to disk. It has access to an in-memory intermediate layer abstraction called buffer pool. High level architecture of how data is accessed by the execution engine The execution engine requests pages from the buffer pool by providing a page identifier. It can read the content of the page and send the data to the client. Or it can modify the content of the page and send it back to the buffer pool so that the change can be persisted. The execution engineaccess the buffer pool thinking that it is accessing the disk.

I really think the term buffer pool is kind of misleading here, and it confused me a lot. And that’s because the purpose of this structure is not only to manage a pool of empty buffers and its reuse but also it is responsible for caching the disk pages into memory for future reuse (not only by the requesting thread but also for concurrent threads). And as with any cache, also responsible for caching replacement policy. For this reason, a more appropriate name for the buffer pool is page cache. With this in mind, you can imagine the importance of the buffer pool for speeding up the database access to disk.

The buffer pool can be implemented with a statically allocated array of frames. Frames start empty and fill up as pages are being requested. There is an associative structure called page table, that indicates which page is being held by each frame. A representation of a full buffer pool The above image indicates the two data structures used for implementing a buffer pool: a frame array and a page table. The image represents a buffer pool that can cache a maximum of 4 pages. The pages being cached are 4, 5, 6, and 7. The arrows indicate where the page can be found in memory (in the array of frames). Let’s look into another buffer pool: A representation of a buffer pool with one empty frame At this, we can see that there are 3 pages in cache, and one space left. The frames that are free and are ready for receiving a new page are tracked in an additional structure called free list.

The pages in red indicate the pages that are being accessed by some thread. That indication is very important because it means that the page cannot leave the buffer pool until all accesses are finished. We will be back to this.

Another important concept is page replacement. What happens when the cache is full, and a page that is not in the cache is requested? We need to evict a page that is not being accessed and give its frame to the requested page. There are many algorithms (or policies) to decide which page to evict. We’ll talk about this later.

With these concepts in mind, here is how we can represent our buffer pool in Go code

//BufferPoolManager represents the buffer pool manager
type BufferPoolManager struct {
    diskManager DiskManager
    pages       [MaxPoolSize]*Page
    replacer    *ClockReplacer
    freeList    []FrameID
    pageTable   map[PageID]FrameID
}

//NewBufferPoolManager returns a empty buffer pool manager
func NewBufferPoolManager(DiskManager DiskManager, clockReplacer *ClockReplacer) *BufferPoolManager {
    freeList := make([]FrameID, 0)
    pages := [MaxPoolSize]*Page{}
    for i := 0; i < MaxPoolSize; i++ {
        freeList = append(freeList, FrameID(i))
        pages[FrameID(i)] = nil
    }
    return &BufferPoolManager{DiskManager, pages, clockReplacer, freeList, make(map[PageID]FrameID)}
}

The buffer pool receives a DiskManager implementation and a replacer. In our case, we are going to implement the clock replacement policy, implemented by ClockReplacer, for page eviction. Internally, pages represents the frame array of fixed size (MaxPoolSize); pageTable is a map the indicates which frame holds which page; and freeList tracks the array of free frames. Initially, we have all frames ids inside the freeList, because all frames are free. That also means all frames pointing to nil.

And here is the API provided by our buffer pool implementation:

NewPage() *Page
FetchPage(pageID PageID) *Page
FlushPage(pageID PageID) bool
FlushAllPages()
DeletePage(pageID PageID) error
UnpinPage(pageID PageID, isDirty bool) error

NewPage() *Page: a clean new page can be requested. The buffer pool will ask the disk manager for a new page id;
FetchPage(pageID PageID) *Page: fetches a known page. If the page is in cache, it is immidiatatly returned. If not, the buffer pool will find it on disk and cache it;
FlushPage(pageID PageID) bool: writes the page that is in cache to disk;
FlushAllPages() writes all pages in cache to disk;
DeletePage(pageID PageID) error: deletes page from cache and disk;
UnpinPage(pageID PageID, isDirty bool) error: indicates that the page is not used any more for the current requesting thread. If no more threads are using this page, the page is considered for eviction.

How to represent a page

We are going to represent our page in a very simple manner. Our focus here is to learn how the buffer pool works. Our page is just a []byte followed by an id. There is some metadata that needs to be tracked. One is the pinCount that tracks the number of concurrent accesses. Each time the page is accessed the pinCount is increased. When it is no more in use, the pinCount is decreased. So, if the pinCount is greater than 0, we say the page is pinned and it is indicated in our figures by the red mark. The other metadata variable is isDirty, which indicates that the page has been modified after it was read from disk.

// PageID is the type of the page identifier
type PageID int

const pageSize = 5

// Page represents a page on disk
type Page struct {
    id       PageID
    pinCount int
    isDirty  bool
    data     [pageSize]byte
}

The Clock Replacement algorithm

When the buffer pool is full, and a new page needs to enter the cache, some page needs to be chosen to leave the cache. This is a classic cache eviction problem and there is no perfect solution because predicting which pages are going to be requested is a complicated problem. If you can’t predict which pages are going to be requested, you don’t know which pages must remain in cache and which pages must be evicted. The best you can do is to reduce the number of wrong page evictions. There are many strategies like FIFO (first-in first-out), LRU (least recently used), and LFU (least frequently used). We are not going to explore all of them and their trade-offs. For our buffer pool manager implementation, we have chosen one policy called CLOCK.

When a thread from the execution engine no longer needs a page, it calls the UnpinPage method from the buffer pool. This method reduces the pinCount of the page, and if the pin count reaches zero, the frame id of where that page allocated is stored in the Clock Replacer for future analysis. If, anytime, the same page is accessed again (its pin count increases), and it needs to be removed from the Clock Replacer.

The Clock Replacer keeps tracks of all pages that are not being accessed in a circular buffer. For each page, it also keeps a reference bit to indicate if the page was recently accessed. There is a Clock Hand or Clock Pointer (CP) that iterates through the circular buffer looking for a victim. Clock Replacer representation The Clock Replacer provides the following API:

func Victim() *FrameID
func Unpin(id FrameID)
func Pin(id FrameID) 

Unpin is called when the buffer pool wants to make a page eligible for eviction because it is no longer accessed;
Pin is called when the buffer pool wants to remove a page from cache eviction election because it was accessed and it cannot be evicted;
Victim elects a victim through the clock replacement policy.

Let’s see how the clock replacement policy works using an example.

The state of the Clock Replacer after the calls Unpin(1), Unpin(2) and Unpin(3).

num	ref	clock hand
1	1	<-
2	1
3	1

We have 3 frames eligible for eviction. When Victim() is called, the clock replacement algorithm is run. The clock hand iterates through the circular buffer checking the reference bit. If the bit is set to 1 (that means that the frame was recently added to the clock replacer), it changes the bit to 0. Below is how the clock replacer state changes when Victim() is run until it finds a victim for eviction.

num	ref	clock hand
1	0
2	1	<-
3	1

num	ref	clock hand
1	0
2	0
3	1	<-

num	ref	clock hand
1	0	<-
2	0
3	0

num	ref	clock hand
2	0	<-
3	0

If we call Unpin(4), we have

num	ref	clock hand
2	0	<-
3	0
4	1

And after a call to Victim(), 2 is evicted.

num	ref	clock hand
3	0	<-
4	1

Fetching a page

First, we check if the page is in the buffer pool. If we can find the page in the cache, we increase the pin count (to indicate there is one more thread accessing the page) and make sure it not in the clock replacer by calling Pin.

if frameID, ok := b.pageTable[pageID]; ok {
    page := b.pages[frameID]
    page.pinCount++
    (*b.replacer).Pin(frameID)
    return page
}

If it is not in the buffer pool we need to find a frame to allocate this page. The buffer pool can have free frames or it can be full. The getFrameID method handles this. We check our freeList to see if there are any frames available. If we can find an empty frame, we return its id and remove it from freeList. If we can’t find an empty frame, we have to elect a frame from the buffer pool to free space by running the clock replace algorithm. The bool indicates if the frame id came from the free list or from the eviction algorithm.

func (b *BufferPoolManager) getFrameID() (*FrameID, bool) {
    if len(b.freeList) > 0 {
        frameID, newFreeList := b.freeList[0], b.freeList[1:]
        b.freeList = newFreeList

        return &frameID, true
    }

    return (*b.replacer).Victim(), false
}

Now that we have a frame id and it didn’t come from freeList, we have to remove the page that is allocated in that frame. But if the page was modified (isDirty set to true), then we have to first write it to disk.

if !isFromFreeList {
    // remove page from current frame
    currentPage := b.pages[*frameID]
    if currentPage != nil {
        if currentPage.isDirty {
            b.diskManager.WritePage(currentPage)
        }

        delete(b.pageTable, currentPage.id)
    }
}

Finally, we read the page from disk and allocate it to the chosen frame id

page, err := b.diskManager.ReadPage(pageID)
if err != nil {
    return nil
}
(*page).pinCount = 1
b.pageTable[pageID] = *frameID
b.pages[*frameID] = page

Requesting a new page

Requesting a new page follows the same logic as fetching a page. We need to make sure there is space for a new page in the buffer pool by checking the free list or by replacing a page. After that, the buffer pool requests the disk manager a new page and put it in the cache.

// allocates new page
pageID := b.diskManager.AllocatePage()
if pageID == nil {
    return nil
}
page := &Page{*pageID, 1, false, [pageSize]byte{}}

b.pageTable[*pageID] = *frameID
b.pages[*frameID] = page

Indicating that a page is no longer accessed

After some work is done by a thread on a page, it needs to call the UnpinPage method to indicate the buffer pool that it not using that page anymore. This call is important because it lets the buffer pool decide if that page can be considered for eviction.

The method decreases the pin count of the page. If the pin count reaches 0, it means that there is no thread accessing that page. So, that page is added to the clock replacer to be considered for eviction.

page := b.pages[frameID]
page.DecPinCount()

if page.pinCount <= 0 {
    (*b.replacer).Unpin(frameID)
}

A visualization tool

I was not satisfied with only implementing the buffer pool. I wanted to able to visualize it working. So I went on a mission to build a small JavaScript application that could interact with the buffer pool. I built a web server that instantiated a buffer pool and could receive HTTP requests and send commands to the buffer pool.

Here is a small demo: Buffer pool demonstration You can see four initial calls of New Page, loading all frames of our buffer pool. We flush all the pages to disk. Flushing the pages to disk reduces the pin count of all pages by one (that’s why they go from red to black). After that, Unpin is called to pages 2 and 3, adding both respective frames to the clock replacer. Delete is called for page 2, removing it from the clock replacer, the buffer pool, and the disk, causing a disk fragmentation. New page is called, and page 4 enters the buffer pool in the free frame 2. New page is called again, but the buffer is full. The clock replacement algorithm runs and frees frame 3 for page 5, evicting page 3. Flush all is called writing pages 4 and 5 to disk. Fetch Page is called on page 0 twice, increasing the pin count by two. That means that the Unpin needs to be called twice, in order to make it eligible for eviction. Last, New Page is called, but the buffer is full again. After the clock replacement algorithm is run, page 6 goes to frame 0.

You can play yourself by following the instructions at the repo.

Final thoughts

I have been studying databases for awhile mainly through the course Intro to Database Systems of Andy Pavlo. In the course, he talks about how mmap is most of the time a bad idea for solving the “data in disk larger than the available memory” problem and introduces the concept of buffer pool. One of the assignments is to implement a buffer pool manager in C++. That’s where the idea of implementing one in Go came from.

Even though mmap and buffer pool are two completely different implementations to solve the same problem, I have come to found both very similar. The buffer pool acts like it has all pages in memory, and if the page is not in the cache, it removes one from the cache and fetches the page from the disk putting it on the cache. Pretty much the same logic of the virtual memory that is behind the mmap call. The difference in the buffer pool approach is that everything is in control. The way pages are flushed, locked, and evicted from cache is controlled by the buffer pool. And this kind of control, you completely lose with mmap. And that is why is not to be recommended unless you really know what you are doing.

A buffer pool is supposed to be a shared resource and it is meant to be used by multiples threads doing queries at the same time. In this implementation, it was not our focus to build a thread-safe buffer pool protecting the data structures from race conditions. A thread-safe implementation is planned.

I am on a journey to learn more about databases by implementing parts of one. If you like this kind of content, follow me on Twitter. I will be sharing more related stuff.

Full code at brunocalza/buffer-pool-manager.

Learn more

References

68.6. Database Page Layout - PostgreSQL documentation on how database pages are laid out in storage. ↩
14.8.1 InnoDB Startup Configuration - MySQL documentation on InnoDB’s page size configuration. ↩

But how, exactly, databases use mmap?

2021-01-18T00:00:00+00:00

In a previous post Discovering and exploring mmap using Go, we talked about how databases have a major problem to solve, which is: how to deal with data stored in disk that is bigger than the available memory. We talked about how many databases solve this problem using memory-mapped files and explored mmap capabilities.

Knowing that databases use memory-mapped files to solve the problem was not enough for me. It solved part of the mystery but a question remained: how, exactly, databases use mmap to read and write data from disk?

I decided to dig through a database source code to answer that question. There are plenty of databases that use mmap. Some of them decided to not use anymore. Some examples: SQLite has an option of accessing disk content directly using memory-mapped I/O¹, it seems LevelDB used to use but it changed it², Lucene has an option with MMapDirectory³, LMDB uses mmap⁴, a simple key/value in-memory database from Couchbase called moss uses mmap for durability of in-memory data⁵ and MongoDB removed mmap storage engine for WiredTiger⁶.

I chose bolt, a simple key/value store implemented in Go by Ben Johnson and inspired by the LMDB project, for this endeavor. Mostly because of source code simplicity and my familiarity with Go language. I know a simple key/value store might not be the most complete source code for learning all the details of reading/writing data to disk, but as I have found out, it was more than enough to get a grasp of it.

The original bolt repository is no longer maintained. A fork of bolt called bbolt is maintained and used by etcd⁷. If you are not familiar with bolt, I recommend the articles Intro to BoltDB: Painless Performant Persistence⁸ and Bolt — an embedded key/value database for Go⁹.

The start

I download the code to my machine and opened it in my editor. I thought a good place to start digging was to find out where the database was initialized and look for any references of mmap there. Like most embedded databases, bolt has an Open method for opening the database or creating a new one if it does not exist. Inside it, I found a reference to a private mmap function. That’s a good start.

// Memory map the data file.
if err := db.mmap(options.InitialMmapSize); err != nil {
    _ = db.close()
    return nil, err
}

How much memory should I allocate?

The private mmap is responsible for opening the memory-mapped file. In order to do this, it needs to figure out how much memory it is going to allocate. This task is accomplished by another method called mmapSize. Given the size of the database, this method figures out how many bytes of memory should be allocated.

It starts by doubling the size from 32KB to 1GB. But if the database is larger than 1GB, it grows 1GB at a time.

// Double the size from 32KB until 1GB.
for i := uint(15); i <= 30; i++ {
    if size <= 1<<i {
        return 1 << i, nil
    }
}

...

// If larger than 1GB then grow by 1GB at a time.
sz := int64(size)
if remainder := sz % int64(maxMmapStep); remainder > 0 {
    sz += int64(maxMmapStep) - remainder
}

It is all fine by now. That’s how bolt figured out how much to allocate. But there is a piece of the puzzle now that will be very important when we talk about database storage layout. After figuring out how much to allocate, it needs to ensure that the allocated size is a multiple of the page size. If you are not familiar with database storage and don’t know what a page is, don’t worry, we’ll be back to this.

// Ensure that the mmap size is a multiple of the page size.
// This should always be true since we're incrementing in MBs.
pageSize := int64(db.pageSize)
if (sz % pageSize) != 0 {
    sz = ((sz / pageSize) + 1) * pageSize
}

Shouldn’t it check if we are allocating more than we have available? That’s the last piece of the mmapSize method.

// If we've exceeded the max size then only grow up to the max size.
if sz > maxMapSize {
    sz = maxMapSize
}

The maxMapSize constant is set to 0xFFFFFFFFFFFF on AMD64 architectures.

The AMD64 architecture defines a 64-bit virtual address format, of which the low-order 48 bits are used in current implementations. This allows up to 256 TiB (248 bytes) of virtual address space. - x86-64 Wiki

This is the limit of the bolt database file.

Calling the system call

Now that bolt knows how much it should allocate, it calls the system call mmap. Here is the full code for Unix-like environments:

// mmap memory maps a DB's data file.
func mmap(db *DB, sz int) error {
    // Map the data file to memory.
    b, err := syscall.Mmap(int(db.file.Fd()), 0, sz, syscall.PROT_READ, syscall.MAP_SHARED|db.MmapFlags)
    if err != nil {
        return err
    }

    // Advise the kernel that the mmap is accessed randomly.
    if err := madvise(b, syscall.MADV_RANDOM); err != nil {
        return fmt.Errorf("madvise: %s", err)
    }

    // Save the original byte slice and convert to a byte array pointer.
    db.dataref = b
    db.data = (*[maxMapSize]byte)(unsafe.Pointer(&b[0]))
    db.datasz = sz
    return nil
}

It is a pretty straightforward code. I have some observations to make about syscall.PROT_READ, but I’ll leave for the next session. It is nice to see the call to madvise there, although I don’t know its benefits¹⁰. I would love to know the importance of that call, if it improves performance or if another flag could be set to get different behavior from OS for different use cases.

The mapped memory is set to the variables db.dataref and db.data.

db.dataref = b
db.data = (*[maxMapSize]byte)(unsafe.Pointer(&b[0]))

I would like to know about the importance of keeping track of both variables. I could not grasp what is going on in the conversion to db.data¹¹. But anyway, what we have to keep in mind is that is through these variables that bolt will read data from disk.

What about writes?

While skimming through the source code, I looked for evidence of how mmap was used for both reads and writes. I dug both Get and Put method. I could not find any place where the references to db.dataref or db.data were being updated. I discovered that the writes to disk happen when Commit is called to a transaction. But there I could only find calls to WriteAt¹². So I gave up my search of trying to understand how mmap was used for writes.

Then, suddenly, while looking back to the call of mmap, I noticed the syscall.PROT_READ flag that I have not noticed the first time I looked at the code. So mmap, is only used for reads in bolt. Another place that indicates this is in the definition of DB struct:

dataref  []byte   // mmap'ed readonly, write throws SEGV
data     *[maxMapSize]byte

That made perfect sense to me. Since flushes to disk are very hard to control when using mmap¹³, it is probably the safest approach. How bolt does writing is a topic of another post.

How the database file is structured?

We know how and when bolt allocates memory and that mmap is not used for writes. But how, exactly, bolt can find the value of a key? To understand that, we have to understand how typically databases structure their files. I am not going to do deep here. Mostly because I don’t understand enough to go deep. Just going to try to give a glimpse of what is going on.

A file is just an array of bytes. We have to apply some reasoning to this array of bytes to work with it effectively. Databases structure their files in disk into blocks (chunks of bytes) called pages. bolt is no different. The database file can be seen as Each page has a fixed length of bytes, typically the same size as the OS page (usually 4096 bytes)¹⁴. Here is the part of bolt that sets the pageSize.

// default page size for db is set to the OS page size.
var defaultPageSize = os.Getpagesize()

Every database has its own page layout. The page layout of bolt is defined at page.go as

const (
    branchPageFlag   = 0x01
    leafPageFlag     = 0x02
    metaPageFlag     = 0x04
    freelistPageFlag = 0x10
)

const (
    bucketLeafFlag = 0x01
)

type pgid uint64

type page struct {
    id       pgid
    flags    uint16
    count    uint16
    overflow uint32
    ptr      uintptr
}

id: it is the page identifier used to index the page. Given a page id, I can locate it in the disk through mmap, since the disk file is just a list of continuous fixed length pages;
flags: it tells the page type. There are four types of page: meta, freeList, leaf and branch;
count: indicates the number of elements stored in the page;
overflow: represents the number of subsequent pages;
ptr: indicates the end of page header and start of page data. This is where the keys and values are going to be stored.

A visual representation of a page: Layout of a page With this in mind, we can look at the code that retrieves a page given its id.

// page retrieves a page reference from the mmap based on the current page size.
func (db *DB) page(id pgid) *page {
    pos := id * pgid(db.pageSize)
    return (*page)(unsafe.Pointer(&db.data[pos]))
}

How to perform an efficient search?

We know how the database file is structured in disk and we know how to retrieve a page from disk. But how a bucket.Get([]byte("key")) search works? We are not going to go into too much detail here. I hope the abstraction I created will be enough to get a clue about what is going on.

What if the pages themselves, in the data part, contained references to other pages? And what if these references build up to form a B+Tree?

That is exactly what bolt does. Thinking of the page as a node of a B+Tree¹⁵. In a B+Tree we have internal nodes and leaves. That’s the reason for the flags attribute, to indicate what kind of node that page is. Abstract representation of a B+Tree in bolt So to perform the search of a key, you start at the root node a do a B+Tree traversal on mmapped disk pages. Therefore, bolt is a memory-mapped B+Tree file. The more memory you have, the more it will behave like a memory key/value store.

There are many more details about this process. Most of it I don’t understand myself. So let’s just keep it simple at this abstract level.

Resizing mmap

When I started looking at the source code, I searched for all calls of mmap. The first was found at the Open method, as explained in the beginning. And the second one was found at the allocate method.

When bolt is writing it needs to make sure it is not consuming all the allocated memory. If it sees that it is going to exceed the database size, it resizes the memory.

// Resize mmap() if we're at the end.
p.id = db.rwtx.meta.pgid
var minsz = int((p.id+pgid(count))+1) * db.pageSize
if minsz >= db.datasz {
    if err := db.mmap(minsz); err != nil {
        return nil, fmt.Errorf("mmap allocate error: %s", err)
    }
}

Conclusion

Reading bolt source code is a very nice way of understanding database internals. It was not very intimidating as I thought it would be. We ignored most of its source code, trying to focus only on how mmap is used to retrieve data from disk efficiently. There are so many more concepts we can learn from bolt like transactions, atomic, isolation, concurrency control, but I’ll leave for another posts.

It is important to remind that the strategy used by bolt is just one of multiple strategies. Others databases uses different page layouts and different data structures. However, on a higher level the logic of mmapped databases should be the same I guess.

I am on a journey to learn more about databases. If you are interested you can follow me on twitter, where I share more related content.

If you want to learn more about bolt

References

SQLite Documentation. Memory-Mapped I/O. SQLite.org. Accessed January 2021. ↩
Daly, S. (2011). mmap based writing vs. stdio based writing. LevelDB Google Groups discussion thread. ↩
Prante, J. (2012, July 26). Memory-mapped files with Lucene: some more aspects. Jörg Prante’s Blog. ↩
Symas Corporation. Performance Tradeoffs in LMDB. Lightning Memory-Mapped Database Documentation. ↩
Schoch, M. (2017). Building a High-Performance Key/Value Store in Go. GopherCon 2017 presentation. YouTube. ↩
MongoDB Documentation. Storage Engines. MongoDB Manual v4.4. ↩
etcd-io. bbolt: An embedded key/value database for Go. GitHub repository. Fork of Ben Johnson’s Bolt. ↩
NPF. (2014, July 7). Intro to BoltDB: Painless Performant Persistence. npf.io. ↩
ProgVille. Bolt — an embedded key/value database for Go. ProgVille Blog. ↩
Stevens, W. R., & Rago, S. A. (2013). Advanced Programming in the UNIX Environment (3rd ed.). Addison-Wesley Professional. The madvise() system call provides advice to the kernel about the expected usage pattern of the memory region. ↩
Go Documentation. unsafe.Pointer. The conversion creates a typed pointer to the memory-mapped region for direct memory access. ↩
Go Documentation. os.File.WriteAt. Package os documentation for file operations. ↩
Love, R. (2013). Linux System Programming (2nd ed.). O’Reilly Media. Chapter on memory mapping discusses the challenges of controlling when mmap’d data is flushed to disk. ↩
Silberschatz, A., Galvin, P. B., & Gagne, G. (2018). Operating System Concepts (10th ed.). Wiley. The standard page size on most systems is 4KB (4096 bytes). ↩
Cormen, T. H., Leiserson, C. E., Rivest, R. L., & Stein, C. (2009). Introduction to Algorithms (3rd ed.). MIT Press. Chapter on B-Trees and B+Trees. ↩

Discovering and exploring mmap using Go

2021-01-10T00:00:00+00:00

Recently I’ve come to know the concept of memory-mapped files while watching a lecture of the course Intro to Database Systems of Andy Pavlo on database storage. One of the main problems a database storage engine has to solve is how to deal with data in disk that is bigger than the available memory. At a higher level, the main purpose of a disk-oriented storage engine is to manipulate data files in a disk. But if we assume that the data in the disk will eventually get bigger than the available memory, we cannot simply load the whole data file into memory, do the change, and write it back to disk.

This is not a new problem in Computer Science. When operational systems were being developed in the early 1960s, a similar problem was faced: how can we run programs stored in disk that are larger than the available memory? A solution to this problem was made by a group in Manchester, implemented on the Atlas Computer, in 1961. It was called virtual memory. The virtual memory gives a running program the illusion that it has big enough memory, despite the fact that the computer does not have enough.

We are not going to go deep on how virtual memory works. Just have in mind that when a program is accessing memory it is accessing the virtual memory. And maybe the data the program is trying to access is not actually in memory, but it does not matter. The operational system will make pretend that it is by going to disk, and putting it there, and replace an old chunk of memory that is not going to be used.

So, one of the ways a database storage engine can solve the larger than memory problem is to make use of virtual memory and the concept of memory-mapped files.

In Linux, we can make this use by using the system call mmap that lets you map a file, no matter how big, directly into memory. If your program needs to manipulate the file, all it needs is to manipulate the memory. The operating system handles the writes to disk for you.

In some occasions, programmers find this method more convenient than the usual system calls: open, read, write, lseek and close.

A simple demonstration

Here is a small example of how you can take advantage of this in Go using the package mmap-go:

package main

import (
    "os"
    "fmt"
    "github.com/edsrzf/mmap-go"
)

func main() {
    f, _ := os.OpenFile("./file", os.O_RDWR, 0644)
    defer f.Close()
    
    mmap, _ := mmap.Map(f, mmap.RDWR, 0 )
    defer mmap.Unmap()
    fmt.Println(string(mmap))
    
    mmap[0] = 'X'
    mmap.Flush()
}

The beauty is that we could have a much bigger file, and the solution would still work. We would not have to worry about managing memory in order to avoid it filling up.

Detailing mmap capabilities

We’re going to explore more mmap functionalities from the point of view of the API provided by mmap-go. There are probably more features that the native syscall provides that this library does not implement.

The `prot` argument

Here is the mmap.Map signature

func Map(f *os.File, prot, flags int) (MMap, error) 

Let’s look at prot first. The prot argument lets you specify the protection levels of your mapping: RDONLY, RDWR, EXEC are the options provided for mmap-go. These levels are pretty straightforward, RDONLY means you can only read from the mapping, RDWR means you can also write, and EXEC means you can execute code on that mapping. Here is the description of prot from the Linux man:

The prot argument describes the desired memory protection of the
mapping (and must not conflict with the open mode of the file).
It is either PROT_NONE or the bitwise OR of one or more of the
following flags:

PROT_EXEC
    Pages may be executed.

PROT_READ
    Pages may be read.

PROT_WRITE
    Pages may be written.

PROT_NONE
    Pages may not be accessed.

In the unix package, those flags are: unix.PROT_EXEC, unix.PROT_READ, unix.PROT_WRITE and unix.PROT_NONE.

Experimenting with `PROT_EXEC` flag

I’ve become intrigued by the EXEC flag and wanted to see an example of how that works. I’ve Google and could not find any example. So I tried a search in Github by PROT_EXEC and found a good example in C: MMapExecDemo. I replicated this example in Go using mmap-go.

The first step was to create a function that I wanted to be put in memory by mmap allocation, compile it, and get its assembly opcodes.

I created the inc function in inc.go file

package inc

func inc(n int) int {
    return n + 1
}

compiled it with go tool compile -S -N inc.go, then got its assembly by calling go tool objdump -S inc.o.

func inc(n int) int {
  0x22b                 48c744241000000000      MOVQ $0x0, 0x10(SP)
        return n + 1
  0x234                 488b442408              MOVQ 0x8(SP), AX
  0x239                 48ffc0                  INCQ AX
  0x23c                 4889442410              MOVQ AX, 0x10(SP)
  0x241                 c3                      RET

With this, we can build represent our function in bytes on our code

code := []byte{
    0x48, 0xc7, 0x44, 0x24, 0x10, 0x00, 0x00, 0x00, 0x00,
    0x48, 0x8b, 0x44, 0x24, 0x08,
    0x48, 0xff, 0xc0,
    0x48, 0x89, 0x44, 0x24, 0x10,
    0xc3,
}

We allocate our memory with mmap.

memory, err := mmap.MapRegion(nil, len(code), mmap.EXEC|mmap.RDWR, mmap.ANON, 0)
if err != nil {
    panic(err)
}

In this call, we’re using a more complete function called MapRegion that lets you specify how much memory you are allocating (Map allocates the size of the underlying file) and the offset of the file.

In the beginning, we said that the main purpose of mmap was to create a mapping between a file and memory. But in this call we are not indicating any file. mmap can be used just a regular memory allocator by setting nil to the *os.File argument and mmap.ANON to the flags argument. We will talk about more mmap.ANON. Since we are not mapping any file, the offset is 0.

So we have memory allocated with the same size of our code len(code). Since we set the flag mmap.RDWR, we can copy our code to memory.

copy(memory, code)

We have the code of our inc function in memory. In order to execute it, we have to cast that memory address to a function with a signature that matches the signature of our compiled inc.

memory_ptr := &memory
ptr := unsafe.Pointer(&memory_ptr)
inc := *(*func(int) int)(ptr)

When we call inc, we are executing the code we put in memory. That only works because of the flag mmap.EXEC. If that flag was not set, a segmentation violation would occur.

fmt.Println(inc(10)) // Prints 11

I don’t know if this is a real use case. I just wanted to see what it meant to execute code that you put in memory. And there are probably other ways of achieving the same with regular memory allocation and calls to mprotect.

One question that may come up is: but the code is already in the code variable, can’t we just execute it? No, because the memory static allocated to code is not executable. Can we make it executable? I’ve tried to use mprotect on it but still got segmentation violation.

Here is the full working gist.

The `flags` argument

We can have many processes mapping the same memory region. This argument lets us decide about the visibility of the updates happening in the mapping. There are many flags, and you can check them out at mmap. The important ones are unix.MAP_SHARED, unix.MAP_PRIVATE and unix.MAP_ANON.

MAP_SHARED means that changes to the mapping are visible to all processes and will also occur at the underlying mapped file, although we cannot control when.

MAP_PRIVATE means the changes are private and other processes will not see them. And also, they are not carried through to the underlying file.

MAP_ANON means that there is not going to be a mapped file. It is useful for sub-processes communication with shared memory.

I’ve got confused about the mmap-go library implementation. It only provides the mmap.ANON flag, that we used in the above example. If you want your mapping to be private, you can set the mmap.COPY flag to the prot argument. Anyways, you can always use the flags provided by the unix package implementation.

Locking and flushing

Two other nice methods, Lock and Flush, are provided by the API of mmap-go. The Lock method calls the mlock system call that prevents the mapping to be paged out to disk. And the Flush method calls the msync system call that forces the data in memory to be written to disk. This is a good way to trying to have more control over how and when data is flushed to disk.

Wrapping up

I felt kind of stupid of knowing about mmap after so long. I don’t remember it being brought in my college class. For some reason, I felt amazed by it and its capabilities and decided to dig deeper. I like databases and I’m aiming to get a better grasp of them. This means that mmap cannot go unnoticed from my learning. For future posts, I’ll try to bring about the benefits and drawbacks of using mmap, which projects use it, and what kind of problems it is suited for.

Even though the mmap can be used to solve that database problem we stated in the beginning, and many modern databases use it, Andy Pavlo advocates against it and have three lecture on how to databases, that don’t use mmap, manage data.

If you like this kind of content, follow me on twitter. You may find more related stuff there.

My Thoughts on A Plea for Lean Software

2020-03-22T00:00:00+00:00

A Plea for Lean Software¹ is a classical paper that presents us with some hints of why software increases in complexity and gives us some advice on how to avoid or minimize complexity. Here I present my interpretation of Niklaus Wirth’s ideas in a non linear way adding some personal reflections on top of it.

The amazing thing about this paper is that it was written in 1995, when programs where only a few KB or MB big. And Niklaus Wirth² was already claiming that software growth was getting out of control. It amazes me how accurate his observations still are even 25 years later. As an Engineering Manager, I see a lot of what he’s talking about in my mundane activities and I also see myself acting in a lot of ways similar to what he points out as crucial for software complexity growth.

This paper is famous for the following quote: “Software is getting slower more rapidly than hardware becomes faster”. Wirth attributes it to Martin Reiser.

Wirth starts by pointing two factors for ever-growing software:

Rapidly growing hardware performance
Customers’ ignorance of features that are essential-versus-nice to have

The first factor is very easy to grasp and makes total sense. But I had a little trouble understanding the second one. After some reflection, what I could come up from this claim is that the customers’ lack of understanding of what brings value and what does not bring value provides software vendors the wrong kind of information on what to build. And that happens because the userbase is always making requests about the product that they claim are useful but not necessarily are (some of them may be). If the software vendors don’t think critically on what the user is requesting, it will lead to software complexity. This behavior incentivizes a feature factory schema that doesn’t necessarily add value to the customer. I’d would like to add that not only the customers’ ignorance leads to complexity but also the software vendors’ ignorance to recognize customers’ ignorance and have a clear understanding of what adds value to customers’ life. As a software vendor myself, only the fact of being aware of my ignorance and limitation may be a great start for reducing complexity or avoiding it to grow. But that requires a lot of humility.

Wirth’s talks about what he calls monolithic design. You build a single system to assist a diverse userbase that pays for all features but only use a few or if I may add, each user segment uses a small set of features that is different from the other user segment. That leads to his recommendation that we should be building a thoughtful well designed system core that is extensible, which I think is indeed a good thing. However we all know that software development comes with some constraints. And greatest of all constraints is time. Wirth puts a lot of criticism on time pressure declaring that it discourages careful planning and improving acceptable solutions and on the other end, encourages quickly software additions and corrections. But time may be in fact a real constraint and the pressure sometimes makes sense. The problem is that we don’t know how to identify when it as a real constraint versus we’re just making this up. I know that there are cases that the constraint is real, like developing a website for next World Cup. But in most occasions, the time constraint comes from managers’ anxiety (as manager I recognize this), not well thought sprint timeboxes or simply because we have to. Since it may very hard to distinguish a real time constraint from a fantasy one, a good way of stop fooling ourselves it is to use time as an indicator of when we should be ready shipping a next increment of improvement. And that’s agile. Unfortunately it’s very hard to understand what the next increment of improvement should be and people don’t really understand the increment part. Wirth also elaborates on the importance of iterative improvements, not as an escape from the time pressure as I have elaborated, but as an escape from the hardship that is building software:

Initial designs for sophisticated software applications are invariably complicated, even when developed by competent engineers. Truly good solutions emerge after iterative improvements or after redesigns that exploit new insights, and the most rewarding iterations are those that result in program simplifications

Another point made by Wirth is that complexity promotes customer dependence on the vendor. So there is an incentive to make things complex in order to create a dependency of the customer generating a more stable stream of income. In a way it makes a lot of sense. A good simple product in which the customer can solve all its problems by yourself without any sales or customer support might not lead to a great engagement. But I don’t really think that this practice happens consciously though. Most of the time, I would guess it happens accidentally. Even because customer support and a sales team may not be very cheap or scalable. But I do recognize that some vendors may consider (consciously) complexity as a strategy for customer lock-in.

The paper goes on and details more practical technical advice on how to keep complexity low by presenting a practical case called project Oberon and it finishes with 9 lessons learned from this project. Those are very useful and it is worth checking out. I’d love to know more about software complexity and its root causes and I will probably research more about the topic. For now I’ll finish with what I think is best phrase of the paper:

Good engineering is characterized by a gradual, step-wise refinement of products that yields increased performance under given constraints and with given resources.

References

Wirth, Niklaus. “A Plea for Lean Software.” Computer, vol. 28, no. 2, IEEE Computer Society, Feb. 1995, pp. 64-68. DOI: 10.1109/2.348001. ↩
“Niklaus Wirth.” Wikipedia, The Free Encyclopedia, Wikimedia Foundation. Available at: https://en.wikipedia.org/wiki/Niklaus_Wirth. Swiss computer scientist (1934-2024), designer of Pascal, Modula-2, and Oberon programming languages, and 1984 Turing Award recipient. ↩

Bruno Calza - Blog

Building a Grow-Only Counter on a Sequentially Consistent KV Store

Starting simple

Solving read-modify-write with CAS

CRDT-like solution

On consistency models

Sequential consistency

Does sequential consistency explain our results?

The hacky solution

The non-hacky solutions

Fly.io's Broadcast Challenges

Single-node

Multi-node

Fault tolerance

Efficiency

Topology

Batching

Benchmark

Generating Unique IDs with Raft Consensus

Thinking about the system architecture

Generate implementation

Getting the next value

Initializing our distributed counter

Initializing Raft nodes

Background routine

Communication among the nodes

Running the test

Comparing with Snowflake

Implementing Snowflake Unique ID Generation

When Does C++ Call the Move Constructor?

1st Case - Explicit

2nd Case - Initializing from a temporary object

3rd Case - Returning a local object from a function

4th Case - Passing rvalue to a function

5th Case: Emplacing into standard containers

Scoped Threads in Rust

Writing a simple lexer in Rust

The day I discovered vmtouch

Introduction

Confirming that the OS is caching the SQLite data

Playing directly with the files

Final thoughts

Making a change to SQLite source code

Introduction

Navigating the SQLite source code

Making changes to SQLite

Forking Go’s SQLite driver

Putting everything together

There Are Many Ways To Safely Count

Don’t Do This

The Classic

Using Channels

The Atomic Way

Compare And Swap

What About Float Types?

Final Words

Getting To Know Logical Clocks By Implementing Them

The Clock Synchronization and Ordering Problems

Lamport Clocks

The Algorithm

Visualizing

Total ordering

Limitations

Vector Clocks

The Algorithm

Properties

Wrapping Up

References

What Zero-copy Serialization Means?

The Cache is Full

Defining Cache

The “Cache is Full” Problem

Cache Replacement Policies

Implementing Some Policies in Go

The cache

FIFO

LRU

CLOCK

LFU

Final Considerations

4th Case - Passing `rvalue` to a function

Confirming that the OS is caching the `SQLite` data

The `prot` argument

Experimenting with `PROT_EXEC` flag

The `flags` argument