Index

• Mixture of Experts
• Faster Inference
  • Speculative decoding
  • Prefill decode disaggregation
  • Paged attention
  • Prefix caching
  • KV cache offloading

Mixture of Experts

• Helps scale model size without increasing FLOPs
• Dipatch to expert works with
  • Learned routing (Switch Transformers)
  • Randomly (with capacity cap, GShard)
  • Hashing (baseline, not learned)
• HF blog post on MoE
• MoE is typically applied to MLP layers but can also be applied to attention heads
  • ModuleFormer
  • JeTMoE

Faster Inference

Speculative decoding

• Use a smaller model to predict next k-tokens and then run a batch of k trajectories with the larger model (in parallel)
• Inference is memory bound, running 1 prediction takes the same time as k concurrent ones
• Methods
  • Medusa: multiple light weight prediction heads that predict future tokens
  • Multi-token prediction: predict multiple tokens at each inference step
  • N-gram speculation: based off of current context
  • EAGLE: perdict next hidden state instead of next token and then uses the same head for prediction. Very strong!
  • DSpark: from DeepSeek V4; dynamic verification length, parallel backbone + sequential for dependencies when drafting + per-position confidence predictions.

Prefill decode disaggregation

• Prefill: Compute bound; mostly computation of large matmuls to store in KV cache
• Decode: Memry bound; retrieval and storage of KV cache + some compute
• Longer multi-turn sessions = more resuse of KV cache
• Mostly useful for heterogenous compute (high HBM v/s faster compute)
• This only helps when communication (network, disk, memory etc) overhead is lesser than prefilling locally at the inference node
• vLLM supports disaggregated prefilling with LMCache.

Paged attention

• KV cache is normally store as a contiguous block of memory.
• This is wasteful as not all sequences use up the entire context window.
• This leads to lower effective batch size and memory fragmentation.
• With paged attention, KV cache is stored as smaller pages and retrieved on the fly with a lookup table.

Prefix caching

• Store KV cache of an exisiting query to reduce latency.
• Helps with things such as system prompt, multi-turn conversation.
• Essentially helps reduce prefill with stored caches across multiple requests.

KV cache offloading

• KV cache size grows linearly with sequence length.
• User interactions are mostly sparse, KV caches for all requests don’t need to stay in HBM all the time.
• Unused KV cache can be paged out to host RAM or even persistent storage and then paged in when required.
• vLLM supports offloading with LMCache.

steps

1. power off the Steam Deck
2. hold the volume up and power buttons until you hear a chime
3. you’ll now be in the BIOS
4. from here go to the menu item titled “Setup Utility”
5. once there, navigate to the “Advanced Settings” section
6. here you can choose voltages offset from 0mV to -50mV for the CPU, GPU and SoC
7. I decided to use -50mV offset on both the CPU and the GPU and kept the SoC at -30mV

That’s it, happy gaming!

MLX is an array framework purpose built for Apple Silicon; think NumPy but tuned more for Apple’s hardware. It also has higher level abstractions that provide an interface that’s similar to PyTorch. I wanted to learn MLX, what better way to do than to port the model that started it all for LLMs.

the constituents of a GPT

GPT is a decoder only transformer, meaning it goes from a sequence of embedded tokens and then auto-regressively generates a sequence from this pre-fill. It also uses positional embeddings to encode the position of tokens in a sequence.

Each decomposable block is composed of causal (masked) multi-headed self attention, MLP and pre-layer norms for both. This block is then stacked multiple times in series.

Finally, a layer norm and an LM head transforms the hidden dim back up to the vocab size of the tokenizer. This can then be decoded by treating the outputs as logits.

loading pre-trained weights from HF Transformers

Mostly boring stuff here, this involves loading the model weights from the HF Transformers library and then renaming the state_dict keys into a form that can be used with MLX.

improving generation with sampling

Greedy decoding produces the same sequence by picking the most likely outcome every time. This can be improved by introducing two parameteres temperature and top_k alongside sampling to improve the diversity of generated sequences. The temperature parameter controls the sharpness of the probability distrbution of the tokens whereas the top_k parameter controls for how many of the options are considered while sampling. Setting a temperature of 0 falls back to greedy decoding.

what’s next?

Lots of ideas to try! Here’s a couple of them in no particular order:

• Instruction tuned weights for GPT-2
• Training/fine-tuning GPT-2 for some niche task
• Implementing other LLMs such as Llama

install opencode

curl -fsSL https://opencode.ai/install | bash
opencode --version

Config lives in ~/.config/opencode/.

point it at LM Studio

Fire up LM Studio and load a code-focused model – something like Qwen3 Coder 30B works well if your hardware can handle it. Make sure the local server is running, by default it listens on http://localhost:1234.

Then edit ~/.config/opencode/opencode.json:

{
    "$schema": "https://opencode.ai/config.json",
    "provider": {
        "lmstudio": {
            "npm": "@ai-sdk/openai-compatible",
            "name": "LM Studio (local)",
            "options": {
                "baseURL": "http://localhost:1234/v1"
            },
            "models": {
                "zai-org/glm-4.7-flash": {
                    "name": "glm-4.7-flash"
                }
            }
        }
    }
}

The key bit is @ai-sdk/openai-compatible – LM Studio exposes an OpenAI-compatible API, so this just works.

Launch opencode and run /model to verify. You should see “LM Studio (local)” as an available provider.

why this matters

The nice thing about running everything locally is that your code never leaves your machine, there are no token limits, no API costs, and it works offline. The tradeoff is output quality – local models are getting better fast but still lag behind frontier cloud models for complex reasoning. For code review, refactoring suggestions, and boilerplate generation though, they work surprisingly well.

references

• OpenCode
• LM Studio
• Yoyo Lin’s guide on OpenCode + LM Studio for security review

concise commandline monitoring for containers

import torch

def get_size(t: torch.Tensor):
    return t.numel() * t.element_size()

That’s it!

It was a nice game but it did not really last for as long as I had hoped for.

You can, just add .atom to the original URL of the Gist profile and et voilà!

More verbosely, subscribe to https://gist.github.com/{username}.atom from your feed reader of choice.

Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services.

Firecracker enables you to deploy workloads in lightweight virtual machines, called microVMs, which provide enhanced security and workload isolation over traditional VMs, while enabling the speed and resource efficiency of containers.

Firecracker is a virtual machine monitor (VMM) that uses the Linux Kernel-based Virtual Machine (KVM) to create and manage microVMs.

Firecracker is generally available on 64-bit Intel, AMD and Arm CPUs with support for hardware virtualization. 🤯

Julia Evans has a fantastic post on the same: Firecracker: start a VM in less than a second, you should give it a read!

V8 orchestrates isolates: lightweight contexts that provide your code with variables it can access and a safe environment to be executed within. You could even consider an isolate a sandbox for your function to run in.

A single instance of the runtime can run hundreds or thousands of isolates, seamlessly switching between them. Each isolate’s memory is completely isolated, so each piece of code is protected from other untrusted or user-written code on the runtime. Isolates are also designed to start very quickly. Instead of creating a virtual machine for each function, an isolate is created within an existing environment. This model eliminates the cold starts of the virtual machine model.

That reads suspiciously like threads, doesn’t it? Yes, sort of – it depends on the implementation. This is what Chromium’s Blink has to say:

An isolate is a concept of an instance in V8. In Blink, isolates and threads are in 1:1 relationship. One isolate is associated with the main thread. One isolate is associated with one worker thread.

Anyway, circling back to what Cloudflare workers – I wonder what the implications for security and isolation are across the various isolates? There’s various threads of discussion on this, notably this thread from the orange website. It defers the decision to it depends with process based isolation being preferred when running untrusted code.

Yep, that’s right – there’s CSS in that Atom feed, turns out you can do it.

Here’s another blog posting describing how: https://andrewstiefel.com/style-atom-xsl/.

And now its in action here: https://blog.sayan.page/feed.xml, that’s all!

connect to the postgress database docker container

docker exec -it immich_postgres bash

connect to the postgress shell in the container and load the immich database

psql immich postgres

check which users are currently admin users

immich=# select email,"isAdmin" from users;

promote an existing user to the admin role

UPDATE users SET "isAdmin"='t' WHERE email='email@example.com';

And that’s it!

Install the mitmproxy server on your platform of choice. The instructions on mitmproxy.org are fairly straightforward to follow! After installing, run the mitmproxy command to start the proxy server.

Note the IP address/hostname (let’s call this proxyhost) of the computer that is running the proxy server, by default the server listens on port 8080. Essentially, we should have the proxy server reachable at proxyhost:8080 now for any device on the same network as the proxy server.

configure your iOS device to use the proxy

Head over to Settings > WiFi and then tap on the (i) icon beside the network your are connected to.

See: Screenshot 0

Scroll down to the bottom of the page and then tap on the option that says “Configure Proxy”.

See: Screenshot 1

Use the proxy server IP and port from earlier (proxyhost:8080) and hit save.

See: Screenshot 2

Open Safar on the iOS device and head over to mitm.it. You will be greeted with a page that lists CA certificates for various platforms, tap on the one that is for iOS.

See: Screenshot 3

Tap on allow to download mitmproxy’s CA cert to the iOS device’s certificate store.

See: Screenshot 4

Choose the device that you want to install the CA certificate too. Most likely you would want to choose the iPhone from the list.

See: Screenshot 5

The CA certificate for mitmproxy is now downloaded, we need to head on over to the Settings apps now.

See: Screenshot 6

When you open the Settings app on iOS device, you’ll be greeted with a list item titled “Profile Downloaded”. Tap on that!

See: Screenshot 7

Then tap on the “Install” button on the top right corner.

See: Screenshot 8

After that head on to Settings > General > About and scroll down to the bottom of the page. You will see a menu item titled “Certificate Trust Settings”. Tap on it, in the following page enable the certifcate for mitmproxy.

See: Screenshot 9

profit

That’s it, you will now be able to intercept SSL traffic from your iOS device on the proxy server.

cleaning up

I would recommend disabling or removing the mitmproxy CA certificate when not in use, you can simply toggle it in Settings > General > About > Certificate Trust Settings.

You would also need to disable the proxy server configuration for the network whenever you turn off the proxy server. You can do that by heading over to Settings > WiFi and then tap on the (i) icon beside the network your are connected to. Scroll down to the bottom of the page and then tap on the option that says “Configure Proxy” and then tap on the “Off” option from the list.

caveats

You will not be able to interecept SSL traffic for any application that uses certificate pinning. This is because these applications bypass the system certificate store and have their own trust store for certificates. See https://docs.mitmproxy.org/stable/concepts-certificates/#certificate-pinning on working around certificate pinning.

This is where companies like Markmonitor come into the picture. Their services include:

software intended to protect corporate brands from Internet counterfeiting, fraud, piracy, and cybersquatting

Markmonitor has an impressive clientele involving the likes of companies such as Amazon, Google, Akamai, Alibaba, Baidu etc!

Essentially, think of Markmonitor as a service that reminds you to pay for your domain names before they expire so that you aren’t bullied into someone on the interwebs squatting your domain name. This is quite amusing to me as an individual but it is a sensible service nonetheless that benefits corporations.

I still find it quite hard to wrap my head around this. :)

Appendix

amazon.com

# whois.markmonitor.com

Domain Name: amazon.com
Registry Domain ID: 281209_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2023-05-16T18:56:31+0000
Creation Date: 1994-11-01T05:00:00+0000
Registrar Registration Expiration Date: 2024-10-30T07:00:00+0000
Registrar: MarkMonitor, Inc.

akamai.com

# whois.markmonitor.com

Domain Name: akamai.com
Registry Domain ID: 2906556_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2022-04-11T18:10:46+0000
Creation Date: 1998-08-17T04:00:00+0000
Registrar Registration Expiration Date: 2024-08-16T04:00:00+0000
Registrar: MarkMonitor, Inc.

google.com

# whois.markmonitor.com

Domain Name: google.com
Registry Domain ID: 2138514_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2019-09-09T15:39:04+0000
Creation Date: 1997-09-15T07:00:00+0000
Registrar Registration Expiration Date: 2028-09-13T07:00:00+0000
Registrar: MarkMonitor, Inc.

$ file archive.tar.gz
archive.tar.gz: gzip compressed data, last modified: Sun Dec 24 13:42:15 2023, from Unix, original size modulo 2^32 1372672

list contents of an archive

Use wtf: what the fuck is in that archive?

$ tar wtf archive.tar.gz
- 8<-

extract the archive

Use pfx: please fucking extract the archive!

$ tar pfx archive.tar.gz
- 8<-

MacOS: Ctrl + Shift + Option + Q

Windows: Ctrl + Shift + Alt + Q

This will bring up an overlay dialog with wide range of statistics and debugging information.

Note: This shortcut does not work when you are in fullscreen mode.

Table of Contents

• the __debug__ constant
  • __debug__ in REPL shell
  • __debug__ in scripts

the __debug__ constant

Python has a built-in boolean constant named __debug__.

It is set to True whenever you run a REPL shell or call a script with the python script.py i.e. by default.

It is set to False only when you pass the -O or the -OO flags to the python executable.

These flags are described in the Python manual page as:

-O Remove assert statements and any code conditional on the value of debug; augment the filename for compiled (bytecode) files by adding .opt-1 before the .pyc extension.

-OO Do -O and also discard docstrings; change the filename for compiled (bytecode) files by adding .opt-2 before the .pyc extension.

__debug__ in REPL shell

$ python3
Python 3.9.13 (main, May 24 2022, 21:28:31)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> __debug__
True

$ python3 -O
Python 3.9.13 (main, May 24 2022, 21:28:31)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> __debug__
False

__debug__ in scripts

$ cat script.py
print(__debug__)

$ python3 script.py
True
$ python3 -O script.py
False
$ python3 -OO script.py
False