Sean's Blog

Thoughts on Meditations on Moloch

Sean Pedersen — Thu, 25 Jun 2026 00:00:00 GMT

The scary part of Meditations On Moloch is not that evil people exist. The scary part is that evil outcomes can happen without evil people.

Scott Alexander uses Moloch as a name for bad incentive systems. Not one villain. Not one conspiracy. Not one broken law. A system where each person follows the local pressure in front of them, and the total result is something almost no one individually wanted.

The System Has No Mind

The key move is simple. Alexander asks why civilization produces crime, pollution, war, status races, corruption, fake science, and waste when most people dislike those things.

The easy answer is "bad people did it." Sometimes that is true. Often it is too simple.

A factory owner may dislike pollution, but if filters make his products more expensive, cleaner production may kill the company. A researcher may dislike weak science, but if flashy positive results get published and careful negative results do not, the career path is clear. A parent may dislike school pressure, but if every other parent competes for status schools, opting out can punish the child.

These cases do not need monsters. They need an incentive slope. Once the slope exists, people slide.

Multipolar Traps

Alexander calls these patterns multipolar traps. The phrase means that many agents compete in a system where unilateral virtue gets punished.

If you already think in game theory, the structure is familiar. The prisoner's dilemma shows how two people can both choose a worse outcome because each has a reason to defect. The tragedy of the commons shows how many people can destroy a shared resource while each person acts rationally from their own narrow view.

Moloch is the general version. It is the market that rewards the cheapest provider even when cheapness comes from misery. It is the arms race where nobody wants war, but nobody can safely disarm first. It is credentialism where everyone knows the signal is wasteful, but every individual still needs the signal.

From above, the answer is obvious. Everyone should cooperate. Everyone should stop polluting. Everyone should stop lying. Everyone should stop racing for empty signals.

From inside the system, that advice is often useless. The actor who stops first may lose.

Why Las Vegas Matters

The Las Vegas example in the essay is effective because it avoids the usual moral categories. Vegas is not pure evil. It is also not a sane use of shared human effort.

It is engineering talent, money, water, electricity, land, labor, light, and attention shaped into a machine for stimulation. It exists because many local incentives point that way. Gambling psychology, tourism, regulation, real estate, debt, entertainment, and status all stack into one result.

No central mind needed to ask, "Would this be a good use of civilization?" The question never had to be asked.

That is the unsettling part. Moloch does not need to hate beauty. It only needs beauty to be less competitive than addiction, growth, status, or profit.

This is also why digital systems worry me. Nobody at a platform has to wake up wanting a worse society. The metrics can do the work.

Why Everything Has Not Failed Yet

Alexander does not claim that Moloch always wins. The essay lists several brakes.

First, excess resources hide many failures. A rich society can waste a lot and still stay alive.

Second, physical limits matter. Bodies need food. Buildings need material. Machines need energy. Reality blocks some optimizations.

Third, markets and democracy partly track human preferences. They are not magic, but they do let people push back when a system becomes too bad.

Fourth, coordination mechanisms work. Laws, norms, unions, families, religions, traditions, professional standards, and states can all protect shared values. They make some forms of defection costly.

This is the hopeful part of the essay, but it is not comforting. These brakes are fragile. They need maintenance. They can be gamed. They can be captured. They can become traps of their own.

The lesson is not "coordination is impossible." The lesson is that coordination is a real technology. It must be built, tested, repaired, and defended.

AI Makes the Problem Sharper

The darkest part of the essay is its future-facing claim. Technology can make the traps deeper.

If an economy rewards agents that ignore human values, then better tools may help those agents win faster. If memetics rewards ideas that spread rather than ideas that are true, better media can make worse ideas more competitive. If markets reward attention capture, better models can make manipulation cheaper.

This is where the essay meets artificial intelligence. A badly aimed optimizer does not need hatred to erase value. It only needs a target that excludes the things we care about.

That is also why AI alignment cannot only mean "stop the machine from going rogue." It also has to ask what social incentives shape deployment. A model can be technically aligned with a company objective while still helping build a bad society.

The final fear is superintelligence as pure optimization. Not evil. Worse than evil: empty. A machine, market, or posthuman economy could optimize away consciousness, art, love, and leisure because they are inefficient.

That sounds dramatic. It is dramatic. But the smaller versions already exist.

Gardens Need Walls

Alexander contrasts Moloch with gardens. A garden is a place where values can grow because competition has been limited.

Families are gardens. Good labs are gardens. Monasteries, open-source communities, scientific fields, and small towns can be gardens. They work when norms are strong enough to block the race to the bottom.

But gardens have a hard problem. They sit inside a wider world. If the outside world rewards defection, growth, and aggression, the garden must either defend itself or be absorbed.

This is why I do not read the essay as a call for withdrawal. Walled communities can protect something for a while, but they cannot solve the global problem. Authoritarianism is not a clean answer either. The ruler has incentives too. The court, army, bureaucracy, rivals, and succession problem all reintroduce Moloch through another door.

The better goal is scalable coordination that keeps human values alive under pressure.

Main Takeaway

The essay is useful because it changes the target of blame.

Blaming individuals is sometimes needed. But if the structure rewards bad behavior, replacing people is not enough. The next people will face the same pressure.

The deeper question is always: what does the system reward?

If it rewards attention capture, we get addiction. If it rewards credential signals, we get empty schooling. If it rewards profit without liability, we get pollution. If it rewards speed over safety, we get fragile infrastructure. If it rewards viral spread over truth, we get memetic disease.

Moloch is not a demon outside us. It is the name for value-blind selection pressure inside our systems.

The work is to build institutions that make good behavior survivable. Not just morally praised. Survivable.

That means laws, norms, tools, markets, protocols, and communities that make cooperation cheap and defection expensive. It means designing systems where humane choices are not punished by default.

The open question is whether we can do that fast enough. Technology keeps making optimization stronger. Human values only survive if coordination keeps up.

References

Jung & Naiv: Jörg Baberowski on History, Russia and Democracy

Sean Pedersen — Wed, 17 Jun 2026 00:00:00 GMT

Historians are never nowhere. That is the sharp point in this interview with Jörg Baberowski: everyone looks at the world from somewhere, even scientists.

History Is Written From Somewhere

No one observes the world from a neutral point above it. Not historians, not journalists, not scientists. Everyone thinks, writes, and judges from a position.

That does not make knowledge arbitrary. It makes the performance of pure objectivity dishonest. A scholar earns trust not by pretending to have no standpoint, but by making it visible, showing the evidence, and letting readers test the argument for themselves. The choices stay human: which archive matters, which question is worth asking, which concept frames the facts.

Baberowski aims this at German academic culture, which often trains scholars to write as if the thinker had disappeared. The result is a dry, impersonal style that mistakes absence of voice for neutrality. His point comes close to a defense of honest journalism: do not hide judgment, show how it was formed.

Style matters here. Borrowing from Michel Foucault, Baberowski insists that being right is not enough. Arguments need form: rhythm, images, clarity, a sense of the reader. He contrasts German habits with English and American essay traditions, where the writer is allowed to tell a story. Germans suspect that storytelling manipulates, but a dry academic style manipulates too: it can dress up a subjective argument as a fact handed down by the universe. Style does not replace evidence but strongly influences if and how evidence is heard.

Humans Need Stories

Human beings know they will die, and culture is one way of living with that knowledge. Memory, religion, politics, and historical narratives all help people give shape to lives that would otherwise feel unbearable.

Here he draws on Martin Heidegger and the idea of being-toward-death. You do not have to accept the whole philosophy to grasp the point: people tell stories because bare existence is hard to endure.

Many Western stories follow a familiar line of fall, struggle, and redemption. Christianity offers one version, Marxism another. Several modern ideologies keep the same shape while moving salvation from the afterlife into history. But cultures do not all tell the same story. Some imagine history as progress, others as cycles, endings, or returns. The historian's task is to notice these forms, not smuggle one in as self-evident truth.

This explains his skepticism toward universal answers to the meaning of life. Meaning is not attached to life in advance. People give meaning to what they do. That can produce tolerance, because it shows how many different lives make sense from the inside. It also creates discomfort: a violent person can give meaning to violence, a soldier to war. Baberowski does not resolve this. He separates the inner story people tell about themselves from the judgment others may rightly make about their actions.

Putin, Russia, and Ukraine

The political section applies the same method to Russia. Baberowski describes Putin's system as a black box. Outsiders see the consequences but not the balance of fear, loyalty, ambition, and pressure inside the regime. That makes confident explanations dangerous. He calls Putin a dictator while stressing how little we know about the concrete techniques of rule around him.

What is clearer is the narrowing of public speech. Russia once had more room for critical media and dissent. That space has closed step by step, and the war has accelerated it. A failing war gives the regime a powerful excuse to silence the "enemy within." This matters for history. In an open society, different stories about the past compete. In a closed one, the state decides what the past means. Baberowski links this to Stalinism, not because Putin is Stalin, but because both treat history as an instrument of rule.

On Ukraine, he argues that Putin expected a quick victory and then trapped himself. Once the invasion failed to produce submission, admitting failure became too dangerous. The war also produced the opposite of its goal: it strengthened Ukrainian national identity. One of his darker observations concerns territory. He doubts that occupied regions can simply be restored to what they were. War, flight, and resettlement change the social facts on the ground. Legal claims may stay the same, but populations, institutions, and loyalties do not.

Democracy and Conflict

Baberowski refuses the easy contrast between virtuous liberalism and dangerous populism, and he separates the two. Liberalism protects rights and limits power. Democracy means rule by the people. Representative democracy has always held an oligarchic element, because citizens choose representatives but do not govern directly.

That does not make it false. It makes it tense. People want influence over decisions that shape their lives. When too many decisions move to courts, agencies, European institutions, or technocratic procedures, voting starts to feel pointless. Populism feeds on that frustration. It can be useful when it breaks stale consensus and forces elites to answer real grievances. It turns dangerous when it attacks the constitutional order that keeps conflict peaceful.

This may be the interview's strongest democratic insight: conflict is not the enemy of democracy. A democracy begins to fail when conflict no longer has a legitimate path.

Empathy in Foreign Policy

Near the end, Baberowski criticizes the instinct to answer every political problem with more weapons. He points back to Willy Brandt and the idea that defense and trust-building are not opposites.

This is easy to misread. He does not claim NATO expansion caused the war, nor does he present Putin as a misunderstood partner. His argument is narrower: serious politics requires anticipating how other powers will read your actions, even when their reading is paranoid or wrong. You can hold an aggressor guilty and still ask what fears and incentives shape his next move. That is not appeasement. It is strategic seriousness.

Conclusion

A rich and honest interview from both Baberowski and Tilo Jung. Everyone is a subject, and no one reaches pure objectivity. The best we can do is stay aware of our own biases.

References

Making Local LLM Go Brrr

Sean Pedersen — Thu, 04 Jun 2026 00:00:00 GMT

How to run your local LLM well: fast, reliable and with good quality.

Key metrics:

Prefill speed: prompt/input tokens per second
Decode speed: generated tokens per second
Time to first token (latency)
VRAM usage at target context length
Quality at chosen model/quant/context settings
Concurrency, if serving multiple users

Software (Inference)

Choose the serving stack based on workload:

llama.cpp: best general local path, especially GGUF, CPU, Apple Silicon, and mixed CPU/GPU.
- beelama.cpp: DFlash & TurboQuant in llama.cpp with up to 3x faster generation and 7.5x more KV cache in same VRAM
- ik_llama.cpp: llama.cpp fork with additional SOTA quants and improved performance
vLLM: strong GPU server for batching, throughput, OpenAI-compatible APIs, and production-style serving for modern GPUs.
SGLang: good for structured/agentic serving and high-throughput multi-call workloads for modern GPUs.
ZML: Zig based model run time.
LuceBox: Local LLM inference server built for speed. Custom kernels, speculative prefill & decoding. (very advanced optimizations like hot MoE VRAM cache)
Uzu: A high-performance inference engine for AI models (optimized for Apple hardware)

Performance checklist:

Use the fastest supported attention kernels: FlashAttention, FlashInfer, FlashMLA, etc.
Try speculative decoding / MTP / EAGLE-style decoding when supported, but benchmark with your actual model and sampling settings.
Preserve prefix/KV cacheability:
- keep the system prompt byte-identical
- append new messages rather than rebuilding/changing history
- avoid dynamic timestamps or changing tool schemas in the prompt prefix
- use server-side prefix caching when available
Tune KV cache precision:
- for long context, test q8 KV even with q4 weights
- aggressively quantized KV can hurt long-context coherence
Approximate KV-cache compression for long-context or high-concurrency workloads:
- TurboQuant: developed by Google (blog article)
- KVarN: claims to beat TurboQuant on all dimensions (beellama.cpp supports it)

Tool-calling reliability:

Add middleware for schema repair, retries, validation, and constrained tool loops -> https://github.com/antoinezambelli/forge

TODO:

eval dynamic model routing based on query complexity (fast vs smart model)

Open Models

Run LLM models locally for complete control and privacy. Open-source (reproducible training) vs open-weight (free model weights) models. Dense models are smart but big and slow, Mixture of Expert (MoE) models only compute a fraction of parameters per layer and thus are faster to execute than dense models.

Compare model capability: https://artificialanalysis.ai/models
Find compatible models for your hardware: https://www.canirun.ai/ or try https://github.com/AlexsJones/llmfit -> rule of thumb: plan 70% of VRAM for the model weights and 20% for the KV-Cache.
Community benchmarks for local LLM: https://localmaxxing.com

Most models are too big for consumer GPUs, so quantized versions (compressed parameters) are used. Mixture of Quants (MoQ) is a new very efficient quant variant that does not quant weights uniformly but based on importance.

Curated open model list:

Qwen3.6-35B-A3B Q4_K_XL: Strong MoE (3B active) model with MTP can run even on 8GB VRAM GPU with CPU offloading
- Uncensored version - Reddit thread
- https://x.com/witcheer/status/2053809265538678789
- https://www.reddit.com/r/LocalLLaMA/comments/1tc132c/llamacpp_docker_images_to_run_mtp_models/
- Custom thinking grammar (limit overthinking): https://github.com/andthattoo/structured-cot
  - TODO: find optimal thinking grammar using GEPA
Qwen3.6 27B Q3_K_M - dense model, very good (Sonnet 4.6 performance) can run on 16GB VRAM
Qwen3.5 9B Distilled - small but capable agentic dense model good for <8GB VRAM
- MoQ variant (very efficient quantization)
LFM2.5-8B-A1B - very fast MoE model 1.5B active + 128k context (agentic usefulness is limited though...)
MiniCPM5-1B - optimized for mobile CPU/NPU inference (32k context window)

Hardware (GPU)

VRAM matters more than raw TFLOPs for model & context (prompt) size, but memory bandwidth and tensor cores matter for speed. Used datacenter GPUs can be good value, but check form factor, cooling, power, driver support, and PCIe vs SXM.

Interesting used GPU options:

GPU	VRAM	Bandwidth	TDP	FP32 TFLOPS	FP16 TFLOPS	Notes
Tesla V100 (SXM2)	16/32 GB HBM2	900 GB/s	300W	15.7	125 Tensor	Needs SXM board or riser, check cooling.
Tesla V100 (PCIe)	16/32 GB HBM2	750 GB/s	250W	14.1	112 Tensor	Standard form factor, strong used option.
Tesla P40	24 GB GDDR5X	346 GB/s	250W	12.0	12.0	Lots of VRAM for cheap, no Tensor Cores.
Tesla P100 (PCIe)	16 GB HBM2	732 GB/s	250W	9.5	19.1	Cheap, but old — less attractive than V100/P40.
GTX 1080 Ti	11 GB GDDR5X	484 GB/s	250W	11.3	11.3	Cheap but VRAM-limited, no Tensor Cores.
RTX 3090	24 GB GDDR6X	936 GB/s	350W	35.6	71.2 Tensor	Often the practical local LLM sweet spot.
Intel Arc A770	16 GB GDDR6	560 GB/s	225W	19.7	39.3 XMX	Good llama.cpp SYCL support; get the 16 GB variant.
Intel Arc B580	12 GB GDDR6	456 GB/s	190W	14.4	28.8 XMX	Battlemage arch, better perf/watt than A770, solid llama.cpp support.
AMD BC-250	16 GB GDDR6	448 GB/s	220W	6.9	13.8	Mining card based on PS5 APU, ROCm support varies.

One or two used Tesla V100 16GB cards are the best bang for the buck.

TODO:

Check current AMD ROCm support.
Compare used datacenter GPUs against RTX 3090/4090/5090-class consumer cards.
Benchmark watts/token, not just tokens/sec.

References

#AI #tutorial

Search Engine Optimization

Sean Pedersen — Tue, 02 Jun 2026 00:00:00 GMT

SEO (search engine optimization) is a shitshow of wrong incentives: not good work / content / reputation counts but who can allocate most resources (money and time) to play the SEO game well (place backlinks and publish content). A better system would be a peer to peer based trust network but until then lets play the SEO hunger games using the best available tools and strategy.

The winning strategy:

build backlinks where possible to build domain authority
- ideally gain natural backlinks by publishing interesting content that people naturally mention and link to
identify content gaps in your website with buy intent (buy intent keywords with low ranked competition + ideally high search volume)
- check keyword search volume: https://trends.google.com/trends/explore
monitor your content (blog posts) performance (traffic + revenue / conversion) and adapt / update over time

Measure the right target: SEO success should be measured by revenue contribution.

free backlink checker: https://www.seobility.net/en/backlinkchecker/
free ranking checker: https://tools.searchengineland.com/rank-checker

claude skill: https://github.com/AgriciDaniel/claude-seo

A good backlink is usually a followed, editorially placed link from a relevant and trusted page that itself has visibility or traffic. Nofollow links (from posts on social media f.e.) usually do not pass the same ranking value, but they can still drive referral traffic, brand awareness, and natural discovery.

It seems using country specific top level domains benefits search engine
ranking. https://medium.com/pinterest-engineering/how-switching-our-domain-structure-unlocked-international-growth-e00c8184d5dd

AI tools can help with research, outlines, briefs, clustering, internal-link suggestions, and draft acceleration. Do not publish large volumes of generic AI content without editorial review, original value, and clear usefulness.

Google Search Console is one of the most valuable tools to understand how your website drives traffic.

Google Analytics and Posthog are the most complex and also bloated solutions to track website traffic.

A good open-source alternative (with MCP support):
https://openpanel.dev/

References

https://www.semrush.com/
- https://www.semrush.com/analytics/organic/overview: enter URL of competitors to analyze their keywords
https://ahrefs.com/
automatic content (blog article) + backlink generation
reddit thread on automated SEO
https://www.contentmonk.io/

AI Investment Forecast

Sean Pedersen — Thu, 21 May 2026 00:00:00 GMT

Generative AI is a powerful tool. But the fundamental architecture of large language models imposes structural limits: no autonomy, no taste, no self-awareness. These are not gaps that more compute will close. That gap shapes where the real money should go.

AI Infrastructure

The safest investment assuming centralized AI wins is energy. Giant data centers eat enormous amounts of power. Microsoft signed a long term power purchase agreement that helped justify the restart of Unit 1 at Three Mile Island (now the Crane Clean Energy Center), with the restart carried out by Constellation Energy. Google and Amazon followed with their own nuclear deals.

But decentralized AI cannot be ruled out. The mainframe vs personal computer story is worth remembering. Mainframes won for a while, then PCs took over. Local large language model inference on consumer hardware is already real. If decentralized inference wins, the energy thesis weakens. But chip and hardware makers stay strong either way.

Solid bets on either path:

Rare earth element miners
Chip producers (GPU, RAM, CPU)

Strong bet if centralized AI wins:

Energy companies (nuclear, gas, renewables)

Key risks: Energy companies face regulated returns, grid bottlenecks, and permitting delays. Rare earth miners carry significant China concentration risk, political exposure, and boom-bust pricing cycles. Both are capital intensive with long lead times.

Data Center Infrastructure

The GPU story is well known, but the physical infrastructure behind AI data centers is less discussed and potentially less crowded.

Every new data center requires transformers, switchgear, cooling systems, and grid equipment at scale. Power density per rack is rising sharply with newer GPU generations, and existing infrastructure was not built for it. Many investors believe these physical infrastructure suppliers are more direct beneficiaries of AI capex than generic energy plays.

Investment angle: electrical equipment makers, cooling technology companies, grid infrastructure suppliers.

Key risks: This is cyclical capital goods with long order backlogs. Lead times for transformers and switchgear can stretch years, meaning demand signals today reflect capex plans that could shift if AI investment slows.

Networking

AI clusters are only as fast as the connections between chips. High speed interconnects inside data centers, optical networking between facilities, and data center switching infrastructure are all scaling rapidly to keep pace with compute demand.

This area has attracted less retail attention than GPUs while capturing a meaningful share of AI infrastructure spend.

Investment angle: high speed interconnect suppliers, optical networking companies, data center networking hardware.

Key risks: Networking equipment can be commoditized over time. If AI cluster architectures shift toward wafer-scale or near-memory compute, the interconnect bottleneck changes in character.

NVIDIA Counter Thesis

Chip producers, especially Nvidia, look like a safe bet from any angle. But Nvidia trades at roughly 20 times forward annual revenue. That price assumes years of near perfect execution. The risks are real.

Nvidia's moat rests on three things: CUDA (its proprietary programming framework), its Mellanox interconnect technology, and proven Linux drivers. These have supported extraordinary margins and pricing power in data center accelerators. AMD makes comparable chips on paper but has poor Linux drivers and weak interconnect, which keeps it out of serious data center contention.

Hardware challengers are finding ways around the interconnect problem entirely. Cerebras builds one giant wafer scale chip, eliminating the need for GPU to GPU communication. Its four largest customers are the Mohamed bin Zayed University of Artificial Intelligence (62% of 2025 revenues), G42 (24%), OpenAI, and Amazon Web Services, with the latter two signing in 2026. That OpenAI and AWS, two of Nvidia's largest customers, are buying from Cerebras illustrates how seriously the diversification pressure has become.
Groq uses a deterministic compute architecture that runs inference faster per dollar. Taalas pours LLM directly into silicon, achieving +10K tok/sec. Neither needs to beat Nvidia on raw performance. They just need to be good enough at a fraction of the price. In December 2025, Nvidia and Groq announced an agreement reportedly valued at approximately 20 billion $ to license Groq's AI inference technology and transfer several senior Groq executives to Nvidia, suggesting Nvidia recognized the inference efficiency threat and moved to absorb it rather than outcompete it.

Big tech is building its own chips. Google has been on its sixth generation of in house TPUs since 2016. Amazon has Trainium and Inferentia chips powering clusters of over 400,000 chips for Anthropic. Microsoft, OpenAI, Meta, and Apple all have custom silicon projects underway. Every major Nvidia customer is working to reduce dependence on Nvidia. None of these chips needs to outperform Nvidia. They just need to break even rather than generate extraordinary margins for a rival.

DeepSeek suggested compute may have been massively over provisioned. The Chinese AI lab's reported training efficiency indicated that frontier-adjacent models may require far less compute than many investors assumed. Even if the headline cost figures are incomplete, the broader point stands: algorithmic efficiency can pressure long-term compute demand assumptions. Every major AI lab is now studying how to apply those same techniques. If those efficiency gains prove durable, the compute demand assumptions baked into Nvidia's valuation face real pressure.

Software is reducing CUDA lock in. Frameworks like Triton (OpenAI), MLX (Apple), and JAX (Google) let developers write code once and compile it for multiple hardware targets. This mirrors how hand tuned assembly gave way to C in the 1980s. The CUDA expertise premium will erode as high level abstractions mature.

None of this means Nvidia falls tomorrow. If AI scales as fast as the optimists expect, demand growth could absorb all of these pressures. But at 20 times forward revenue, the market prices in no bad news at all. That is a difficult position to hold.

Robotics: Embodied AI

The potential economic impact of mass scale deployment of robotics is enormous.

The global industrial robot market hit $16.5 billion in 2025. China installed 276,300 industrial robots in 2023, more than the rest of the world combined. By 2024, that number had risen to 295,000, representing 54% of global deployments, with China's operational robot stock exceeding 2 million. The trend is accelerating.

Three things drive it:

Physical AI trains robots in virtual simulations before they touch real environments. Chip makers and robot makers now build dedicated hardware for this. The goal is a "ChatGPT moment" for robots where they learn by experience rather than explicit programming.

Humanoid robots get a lot of attention but remain early. Today they mostly do single purpose tasks in auto plants and warehouses. Whether they become general purpose tools is still open.

Labor shortages are the practical driver. Aging populations in the US, Japan, Germany, China, and South Korea are shrinking workforces fast. Robots fill the gaps. Oxford Economics forecast (published 2019, revisited 2024) projected 20 million manufacturing jobs displaced by robots by 2030. McKinsey estimates AI and robots could automate 40% of US jobs by 2030 while generating $2.9 trillion in economic value per year (via Robotics and Automation News, citing McKinsey research).

That is a massive disruption. The companies that build, program, and maintain these robots win.

Investment angle: robot makers, simulation software, collaborative robot platforms, and industrial automation companies.

Key risks: Robotics hardware carries long sales cycles, high integration costs, and thin margins. Humanoid robots in particular are early; hype currently exceeds deployment. Industrial-grade reliability takes years and significant capital to prove out at scale.

Medicine: Biggest Chance

Drug discovery is where AI might matter most.

Traditional drug development takes 10 to 15 years and costs over a billion dollars per drug. The failure rate is brutal. Most candidate drugs never reach patients.

AlphaFold, Google DeepMind's protein structure prediction model, changed that. It achieved a breakthrough in protein structure prediction, reaching accuracy levels that transformed the field after 50 years of limited progress. AlphaFold's creators Demis Hassabis and John Jumper shared the 2024 Nobel Prize in Chemistry with David Baker, who received the other half for protein design work. AlphaFold can now predict the structure of proteins, RNA, and DNA. This gives drug designers a map of the molecules they want to target.

Isomorphic Labs, the company built on AlphaFold, secured $2.1 billion in funding in May 2026 to push drugs into clinical trials. It has active collaborations with Johnson and Johnson, Novartis, and Eli Lilly. The long term goal is to dramatically shorten the drug discovery process, potentially reducing timelines from many years to a fraction of today's norms.

On the clinical side, a Menlo Ventures survey found healthcare AI adoption jumped 7x in a single year among respondents, with health systems leading at 27% adoption. AI handles triage, scheduling, imaging analysis, and administrative work. The $4.9 trillion US healthcare sector now deploys AI at more than twice the rate of the broader economy.

Investment angle: biotech companies using AI for drug discovery, medical imaging AI, diagnostics, and clinical workflow tools. This sector has real data moats and hard to copy specialized models.

Key risks: Clinical failure rates remain high even if AI accelerates discovery. Finding a promising molecule is only the first step; the path through trials is long, expensive, and uncertain. Regulatory timelines and approval risks are unchanged by AI.

Weapon Manufacturing

AI enabled weapon systems is a growing investment theme in the US and Europe. Autonomous systems, surveillance, logistics optimization, and military robotics are attracting significant government procurement and venture capital. Rising defense budgets in NATO countries and intensifying great power competition are structural tailwinds independent of the broader AI cycle.

Investment angle: defense contractors integrating AI into platforms, autonomous systems developers, drone and counter drone technology companies.

Key risks: Defense procurement cycles are long and unpredictable. Ethics and regulation around autonomous weapons remain unsettled. Government budget priorities can shift, and many defense AI programs remain classified, making competitive positioning difficult to assess from the outside.

Application Layer Risk

Companies with no data moat and a thin app layer on top of general AI face increasing pressure.

When software gets cheap to build, value shifts to the data and relationships underneath. AI lowers software development costs and compresses feature advantages, increasing pressure on application layer businesses whose primary differentiation is functionality rather than proprietary data, distribution, or ecosystem effects. Companies with deep ecosystem lock in, enterprise relationships, compliance advantages, and marketplace network effects are more resilient than those competing purely on features.

Watch companies that:

Sell software AI can replicate fast with no proprietary data or unique pipelines
Lack meaningful switching costs, ecosystem effects, or distribution advantages
Compete on feature parity rather than data or integration depth

Identifying a business as vulnerable is not the same as identifying a good short. Many mediocre businesses stay expensive to short for years. Valuation, balance sheet analysis, and timing matter as much as the competitive thesis.

Ranking of AI Investment Themes

Theme	Confidence	Main risk
Grid and electrical infrastructure	High	Permitting delays, regulated returns
Data center networking	High	Commoditization over time
AI drug discovery	Medium-high	Clinical failure still dominates
Nvidia long-term dominance	Medium	Valuation, custom silicon, inference efficiency
Humanoid robotics	Low-medium	Hype exceeds current deployment
Weapon manufacturing	Low-medium	Procurement cycles, classified opacity
Thin AI apps without moats	High vulnerability	Misidentifying apps with genuine moats as vulnerable

Highest confidence: Data center electrical infrastructure, grid equipment, networking, and custom silicon supply chains. These benefit from AI capex regardless of which AI applications win.

Highest upside, highest uncertainty: AI drug discovery, humanoid robotics, defense autonomy. Long timelines with transformative potential if execution follows.

Most vulnerable: Thin application wrappers without proprietary data or distribution. The addressable market for feature-only software is shrinking as AI commoditizes development.

Most crowded: Nvidia and obvious GPU beneficiaries. The thesis is right, but the price may already reflect it.

References

Top 5 Global Robotics Trends 2025 (International Federation of Robotics)
World Robotics 2025 – Industrial Robots (IFR)
AI Index 2025: Economy Chapter (Stanford HAI)
Isomorphic Labs secures $2.1 billion funding (PR Newswire)
Constellation to Launch Crane Clean Energy Center (Constellation Energy)
NVIDIA Fiscal 2026 Financial Results (NVIDIA Newsroom)
AI and robots in 2025: the robotics revolution (Oxford Economics)
McKinsey: AI and robots could automate 40% of US jobs by 2030 (Robotics and Automation News)
2025: The State of AI in Healthcare (Menlo Ventures)
The Short Case for Nvidia Stock (Jeffrey Emanuel)

#AI #idea

The Human Gut Microbiome

Sean Pedersen — Mon, 18 May 2026 00:00:00 GMT

The gut microbiome is a metabolic organ with no fixed genome. It includes bacteria, archaea, viruses, and fungi. The densest community lives in the colon. It expands digestion, trains immune tolerance, blocks pathogens, transforms drugs, and produces molecules your own cells respond to. The key question is not "which bacteria do I have?" but "what biochemical work can this community do?" Single species narratives mislead. The gut is a community metabolism problem, not a mascot problem.

What microbes actually make

The core function is digesting what you cannot. Fiber, resistant starch, pectins, and inulin reach the colon. Microbes ferment them into short chain fatty acids: acetate, propionate, and butyrate. Butyrate matters most. Colon cells burn it as fuel. It supports the epithelial barrier, dampens inflammation, and modifies gene expression through histone deacetylase inhibition. The gut barrier also works as an immune interface. Microbes compete for space, shape immune signaling, and train tolerance. Without that tolerance, food and harmless microbes start looking like threats. Gut disruption links to systemic inflammation, allergy risk, and metabolic disease. There is no single "healthy" microbiome. Function matters more than taxonomy.

What moves the system

Diet is the strongest everyday input. High fiber and plant rich diets increase fermentation and short chain fatty acid output. Fermented foods add live microbes and their products. But the response is personal. Your starting microbiome shapes how any dietary change lands. Antibiotics are blunt. They damage colonization resistance and let pathogens like C. difficile move in. Recurrent C. difficile is one of the few areas where gut restoration is mainstream medicine. FDA approved fecal microbiota products now exist for it. Probiotics are overmarketed. Effects are strain specific, dose specific, and host specific. Care more about prebiotic food than probiotic branding.

Fecal transplants as a proof of concept

Fecal microbiota transplant is the strongest proof that changing the gut ecosystem can change health outcomes. The clearest case is recurrent C. difficile. Antibiotics damage colonization resistance, C. difficile moves in, and transplanting a healthy donor microbiome closes the gap. FDA approved products now exist for this. The more provocative findings reach further. A small study gave Microbiota Transfer Therapy to 18 autistic children with chronic GI problems. The protocol combined antibiotics, bowel prep, high dose microbiota transfer, and daily maintenance doses. Two years later the children showed maintained GI improvement, higher bacterial diversity, and large gains on autism related rating scales. The trial was small and open label, so it does not prove causation. But it suggests the gut can modulate behavioral and psychological symptoms, at least when GI distress is part of the picture. Depression research tells a similar story. Case reports, pilot trials, and a 2025 meta analysis all point toward short term depressive symptom improvement after FMT, strongest in people with IBS. A bipolar depression trial found no significant difference between donor and autologous FMT, which smells like placebo or procedure effect. The honest summary: FMT for psychiatric conditions is real enough to study seriously, not proven enough to recommend. Donor selection, dose, delivery route, and durability are still open questions.

The gut brain connection

The gut talks to the brain through the vagus nerve, immune signals, microbial metabolites, and the enteric nervous system. Gut state can shift mood, stress response, inflammation, and sleep. That does not make every mental problem a gut problem. The practical bottom line is simple: eat a diverse, plant rich diet. Use fermented foods if tolerated. Avoid unnecessary antibiotics. Be skeptical of tests or supplements claiming to "reset" the gut. Humans are composite organisms. Some biology runs on a shifting microbial system. Modern diet, drugs, and sanitation changed that system faster than evolution could tune for.

References

https://github.com/SeanPedersen/pi-context-skills

Pi - Open-Source Coding Agent

Sean Pedersen — Wed, 13 May 2026 00:00:00 GMT

An open-source coding agent rivaling claude code and openai codex.
Built to be hackable: here is a great intro from the creator.
Vanilla Pi is bare bones and should be configured via extensions - or start with a preconfigured version like oh-my-pi (though oh-my-pi can feel bloated compared to vanilla Pi with its long system prompt, many features and tools). Another intersting open-source coding agent that promises very efficient token usage is Dirac, via hash-anchored edits and AST code edits - check it out.

Install

$ npm install -g @earendil-works/pi-coding-agent

Then run: $ pi
And add your provider with: /login

LLM Providers

Best bang for the buck right now:

OpenCode Go
- Qwen3.7 Plus has best intelligence vs usage limit ratio
OpenRouter: one API to access all big models
Self-host a model on GPU/s

Customize Pi

Prompts

~/.pi/agent/SYSTEM.md : overwrites the global Pi system prompt (be careful)
~/.pi/agent/APPEND_SYSTEM.md : appends to the global Pi system prompt

Pi loads AGENTS.md or CLAUDE.md at startup from:

~/.pi/agent/AGENTS.md for global instructions
parent directories, walking up from the current working directory
the current directory

Packages

Install extensions (packages) from the pi packages marketplace (review code before you use them): https://pi.dev/packages

Your custom extensions live in: $home/.pi/agent/extensions
Installed extensions (packages) are in: TODO

show complete prompt context: https://pi.dev/packages/pi-system-prompt
dynamic workflows (explicit subagent workflow): https://github.com/Michaelliv/pi-dynamic-workflows
sub-agents: https://pi.dev/packages/pi-subagents
plan mode:
- https://pi.dev/packages/@plannotator/pi-extension
- https://pi.dev/packages/pi-pledit
hashline edit: https://github.com/RimuruW/pi-hashline-edit
btw command: https://github.com/dbachelder/pi-btw
goal command: https://pi.dev/packages/@capyup/pi-goal
beads (task management): https://pi.dev/packages/pi-beads-extension
code index / search:
- https://github.com/colbymchenry/codegraph
  - pi extension: https://github.com/SeanPedersen/pi-codegraph
- https://github.com/MinishLab/semble

Skills

Skills are useful to dynamically load relevant documents / tools into the prompt context to solve advanced tasks / use pre-defined complex workflows.

Located in $home/.agents/skills

Contextual Skills

An extension that selects relevant skills per project on first start of Pi (user may edit skills selection later on) - this saves context, as only skills relevant to the each project are in the agents prompt.

Mobile Access

https://www.pipulse.dev/
https://github.com/jademind/pi-statusbar

References

#coding #tutorial #AI

The Ukraine War in Numbers

Sean Pedersen — Thu, 16 Apr 2026 00:00:00 GMT

The Russia-Ukraine war in numbers, as of April 2026 - now past its fourth year. Territory and casualty data from CSIS, Russia Matters, and Al Jazeera. Equipment numbers are photo/video-confirmed minimums from Oryx. Real losses are higher. Frontline tracking via DeepState Map.

Territorial Control

Russia holds about 120,000 sq km of Ukraine - roughly 20% of the country (including Crimea, taken in 2014). The frontline is over 1,000 km long. See DeepState Map for live tracking.

Timeline:

Pre-2022: Russia held Crimea (~7%, ~27,000 sq km) since 2014, plus parts of Donbas
Mar 2022: Peak at ~27% (~165,000 sq km) - Russia pushed toward Kyiv on multiple fronts
Apr 2022: Russia pulled back from Kyiv, Sumy, Chernihiv - dropped to ~24%
Nov 2022: Ukraine took back Kharkiv oblast and Kherson west bank - dropped to ~18%
2023: Summer counteroffensive made small gains (tens of sq km). Little net change
2024: Russia took ~3,600 sq km (+0.6%) - Avdiivka fell Feb 2024, push toward Pokrovsk
2025: Russia took ~4,800 sq km (+0.8%) - biggest yearly gains since 2022
Apr 2026: ~120,000 sq km held (~20%). Recent weeks show slower gains

The Meters-Per-Day War

Russia's advance is extremely slow. CSIS data shows the Pokrovsk offensive - Russia's main push - moves slower than the Battle of the Somme in WWI.

The cost is huge. In 2025, Russia lost an estimated ~415,000 soldiers (killed + wounded) to gain ~4,800 sq km - about 86 casualties per square kilometer. The Pokrovsk push, Russia's main effort since Feb 2024, moved only ~50 km in nearly two years.

Human Cost

Total military casualties estimates (April 2026) have crossed ~2 million (killed, wounded, missing). Multiple independent sources (CSIS, The Economist, Wall Street Journal, Bloomberg) point to similar numbers:

Russia is losing roughly 1,000+ soldiers per day (killed and wounded). On April 15, 2026, Ukraine's General Staff reported 1,010 Russian casualties in a single day. The rate has not slowed - March 2026 was Russia's deadliest month since the invasion began.

Civilians (OHCHR confirmed): 55,600 casualties through Dec 2025 (13,883 killed). 2025 was the worst year for civilians since 2022: 2,514 killed, 12,142 hurt - 31% more than 2024. March 2026 alone: 211 killed, 1,206 hurt (49% jump over February). Real numbers are higher - counting is impossible in occupied areas.

Equipment Losses

Russia has lost 24,440 pieces of military equipment (photo/video confirmed). Ukraine has lost 11,923 - roughly 2:1. Real losses are much higher.

Ukraine has captured 2.25x more Russian equipment than Russia has captured Ukrainian equipment (3,181 vs 1,414).

Armored Vehicle Losses

Armored vehicles (tanks, AFVs, IFVs, APCs, MRAPs) show the biggest gap.

Key ratios:

Tanks: 3.1:1 (Russia loses 3.1 tanks for every Ukrainian tank lost)
IFVs: 4.1:1 (worst ratio - Russia's BMP-series vehicles get destroyed in large numbers during attacks)
AFVs: 4.6:1
APCs: Ukraine loses more (1,346 vs 729) - many Western-donated APCs in use
MRAPs: Ukraine loses more (907 vs 64) - Western MRAPs used heavily near the front

Russia is pulling T-62s and even T-55s from Cold War storage to fill the gap. Oryx has logged 152 T-62M and 10 T-55A losses - 1950s-60s vehicles destroyed by modern weapons.

Artillery and Air Defense

SP artillery losses are close (1,008 vs 840, 1.2:1) - Ukraine's Western howitzers (M109, PzH 2000) are top targets. MLRS losses are one-sided (5.5:1) - Russia has burned through huge numbers of Grad and Uragan systems. SAM losses at 2.3:1 favor Ukraine, whose Western air defense systems remain critical.

Air and Naval

Neither side controls the sky. Russia lost 184 aircraft and 172 helicopters; Ukraine lost 116 and 56. The helicopter gap (3.1:1) comes from Russia using Ka-52 and Mi-28 attack helicopters heavily in assaults. At sea, Ukraine sank the cruiser Moskva and hit Black Sea Fleet ships with Neptune and Storm Shadow missiles, forcing Russia to pull most ships out of the western Black Sea.

Drone Warfare

This war has changed how armies think about drones and robots.

	Russia	Ukraine
Recon UAVs lost	787	563
Combat UAVs lost	21	29
Ground robots (UGVs) lost	73	247

Ukraine lost 3.4x more UGVs than Russia (247 vs 73) - Ukraine uses more ground robots at the front. Both sides now produce millions of FPV drones - cheap one-way attack drones (a few hundred dollars each) that have become the main way to kill armor and infantry, changing the math of war.

The Proxy War: Economics

This is a proxy war between NATO and Russia, fought on Ukrainian soil with Ukrainian lives. The money tells the story.

Russia military spending vs Western aid to Ukraine, by year:

Year	Russia military budget	Western aid to Ukraine	Source notes
2021	~$66B / ~2% GDP	-	SIPRI, pre-invasion baseline
2022	~$75B	~$80B (avg)	Russia: SIPRI. Aid: Kiel Institute avg
2023	~$100B	~$90B (avg)	Russia: gov docs showed >$100B actual
2024	~$140B / ~7% GDP	~$100B (avg)	Russia: SIPRI. Aid ramp-up over time
2025	~$145B budgeted	lower, US aid dropped	German intel estimates real spend ~$250B
Cumulative 2022-2024	~$315B	~$280B	Russia outspent all Western aid combined

Sources: SIPRI, Kiel Institute, Wilson Center. Aid yearly figures are estimates derived from $280 B c u m u l a t i v e a n d k n o w n r a m p - u p p a t t e r n (K i e l r e p o r t s ~$ 90B/year average 2022-2024). Russia stopped publishing realized spending after 2021 - actual spending likely higher than budgeted.

Top aid donors (cumulative through Dec 2024, Kiel Institute):

Russia's war economy: Military budget doubled from ~ $66 B (2021) t o ~$ 140B (2024). Now eats ~30% of the government budget. GDP growth fell to 0.6% in 2025. Manufacturing shrank 7 straight months. Zero Russian tech companies in global top 100.

Ukraine spends ~20% of GDP on defense (up from 3% in 2021) - only possible with Western backing. In 2025, US aid dropped and Europe stepped up (military aid +67%) but didn't fully close the gap. NATO pledged $60B in military aid for 2026.

Why Ukraine?

Why do both sides fight this hard over meters of ground? Zbigniew Brzezinski's The Grand Chessboard (1997) answers with structure, not ideology.

Ukraine decides whether Russia is an empire or a regional power. With Ukraine, Russia gains population, industry, agriculture, Black Sea dominance, and strategic depth. Without it, Russia is boxed into Asia. Brzezinski's most cited line: "Without Ukraine, Russia ceases to be an empire."

Geographic position. Ukraine sits at the intersection of Black Sea access, east/west land routes, and energy transit corridors. Control over Ukraine gives leverage over European security and enables or blocks NATO's eastern coherence. Brzezinski calls this a geopolitical pivot state: not powerful itself, but decisive in who becomes powerful.

Resources multiply position. Ukraine is not just buffer space, it is productive space: among the world's largest grain exporters, significant industrial and metallurgical base, energy transit routes and untapped reserves. Its ports (Odesa, Mykolaiv) control warm water naval access and grain/energy logistics. Russia has Black Sea access without Ukraine (Novorossiysk, Krasnodar coast), but it is narrow, easily bottlenecked, and dependent on Turkey via the Montreux Convention. Ukraine does not create access, it makes access robust.

Preventing a Eurasian hegemon. Brzezinski's thesis: the US must prevent any single power from dominating Eurasia. Ukraine aligned with the West fragments Eurasia. Aligned with Russia, it consolidates it. The proxy war spending above is not charity, it is structural investment.

This is why the war does not end. The strategic payoff is binary, not incremental. Neither side can lose Ukraine without structural decline. Which is exactly what the data shows: enormous casualties, minimal territorial change, relentless continuation. Both sides are playing for the architecture of Eurasia, not for Donbas.

Key Takeaways

Territory costs blood. Russia gained 0.8% of Ukraine in 2025 for ~415,000 casualties. At this rate, taking all of Ukraine would need decades and millions of soldiers.
Record equipment losses. 4,381 confirmed tank losses beat Soviet losses in Afghanistan, Chechnya I, and Chechnya II combined. And these are confirmed minimums.
Slowest major advance in a century. Russia's main offensives move 15-70 meters per day - slower than the Somme in WWI. Huliaipole at 297 m/day is the outlier.
Drones changed everything. $500 FPV drones destroy $3M+ tanks. The cost math of war has flipped.
No air control. Both sides have strong air defense. Flying manned aircraft is extremely risky for both.
Economic pressure building. Russia spends 7% of GDP on military while growth drops to 0.6% and manufacturing shrinks. Ukraine's 20% of GDP on defense only works with outside help.

Possible Outcomes

The data points to prolongation, exhaustion, and partial freezes. Not decisive victory.

1. Prolonged attritional stalemate (most likely)

Russia gained ~0.8% of Ukraine in 2025 for ~415,000 casualties. Advances of 15-70 meters per day are historically slow. Neither side has air superiority. Drones neutralize armor. Breakthroughs are rare. This means years of grinding warfare with shifting front lines but no collapse.

2. Frozen conflict, Korea-style

If Western aid holds steady and Russia's economy keeps straining at ~7% GDP military spending, the war settles into a hard frontline. No treaty, just enforced pause. Ukraine can defend, Russia can advance slowly, neither can end the war at acceptable cost.

3. Negotiated ceasefire under exhaustion

Less likely short-term, plausible long-term. ~2 million military casualties and economic pressure are cumulative. A ceasefire would probably lock in current lines, not restore full territorial integrity. Neither side is winning fast enough to dictate terms.

4. Russian collapse or Ukrainian breakthrough (least likely)

Despite enormous losses, Russia sustains manpower by lowering quality (T-62s, T-55s, mass FPV drones). Ukraine depends on external aid for ~20% GDP defense spending. Without a major shift in technology, alliances, or internal stability, decisive victory is improbable.

Bottom line: This war is about industrial capacity, demographics, and political endurance. Time itself is the main weapon, destroying value faster than territory is gained - hopefully stopping this conflict sooner than later.

References

mRNA Vaccines: How They Work

Sean Pedersen — Fri, 03 Apr 2026 00:00:00 GMT

mRNA vaccines work by mimicking a trick viruses have used for billions of years: injecting genetic instructions (mRNA) into your cells to make them produce a specific target protein.

What is mRNA?

Messenger RNA (mRNA) is a short-lived molecule that carries a protein blueprint copied from DNA. Cells read mRNA through ribosomes, their protein-making machinery, and build the protein the mRNA describes. Once read, the mRNA is degraded. It never enters the cell nucleus and cannot alter your DNA.

The Blueprint Trick

A virus survives by injecting its own genetic instructions into host cells, forcing them to manufacture viral proteins. mRNA vaccines borrow exactly this strategy. The vaccine delivers a synthetic mRNA strand encoding a target protein. Your ribosomes read it and build that protein. Your immune system sees it as foreign, mounts a defence, and stores memory cells for future protection. You are never exposed to the actual pathogen.

Your own cells are the factory. The mRNA is the blueprint.

The Delivery Problem

Naked mRNA degrades within seconds in the bloodstream. It also cannot cross cell membranes on its own. The solution is lipid nanoparticles (LNPs): tiny fatty shells that protect the mRNA and fuse with cell membranes to deliver the payload inside.

LNPs have no precise targeting ability. They distribute largely by proximity. Most are absorbed by cells at the injection site, nearby immune cells, and to some extent the liver, which efficiently clears lipid particles from blood.

Why the Deltoid?

The standard injection site is the deltoid muscle in the upper arm. This is not arbitrary. Muscle tissue in this region is rich in dendritic cells and macrophages, immune cells that pick up foreign proteins and present them to the adaptive immune system. The axillary lymph nodes sit nearby, which is where antibody production and memory cell formation are organised.

Injecting into a vein rather than muscle sends LNPs circulating systemically, reaching organs beyond the intended local site. This distinction between intramuscular and intravenous delivery has practical safety implications and is why injection technique matters. Accidental intravenous injection sends LNPs into blood circulation, potentially reaching the heart and causing myocarditis (inflammation of the heart muscle) like it happened with corona vaccines.

What Happens Inside

LNPs are absorbed by local muscle cells and nearby immune cells
mRNA is released inside the cell
Ribosomes read the mRNA and produce the target protein (for COVID vaccines: the spike protein)
The cell displays fragments of this protein on its surface
The immune system recognises the protein as foreign
B cells produce antibodies; T cells learn to attack cells displaying this protein
Memory cells are stored for long-term protection
The mRNA degrades within days

Beyond Infectious Disease

mRNA technology is not limited to vaccines against pathogens. Its core ability, instructing cells to produce any protein on demand, opens broader applications.

Cancer treatment is the most significant near-term area. Tumours carry mutations that produce abnormal proteins called neoantigens. These are unique to each patient's cancer. Researchers can sequence a patient's tumour, identify its neoantigens, synthesise a personalised mRNA vaccine encoding those targets, and inject it to train the immune system to attack cells displaying them. This approach, already in clinical trials for melanoma and other cancers in combination with checkpoint inhibitors, effectively turns the immune system into a targeted therapy customised per patient.

Other areas in active development include:

Protein replacement therapies: Delivering mRNA that encodes a missing or defective protein, relevant for conditions like cystic fibrosis or certain metabolic disorders.

Autoimmune conditions: Tolerogenic mRNA approaches, where instead of activating an immune response, the goal is to suppress one by instructing cells to express proteins that calm immune activity against self-tissue.

Rare genetic diseases: Periodic mRNA dosing could temporarily restore function of a protein the body cannot produce, without permanent gene editing.

mRNA vs Protein Vaccine — Inflammatory Risk

Activation timeline

mRNA: protein detectable after ~4 hours, peaks at 24 hours, gone by 72 hours
Protein vaccine: antigen immediately available but fixed dose, degrades within hours
Immune response timeline is similar for both (~14 days to meaningful antibodies) — the difference is mechanism, not speed of protection

The amplification tradeoff
mRNA continuously produces fresh antigen for 72 hours, generating a stronger immune signal than a fixed injected protein dose. This is why protein vaccines need adjuvants and often booster shots to match mRNA efficacy — but it also means mRNA keeps the inflammatory stimulus active longer.

Vein exposure risk

mRNA: if accidentally injected intravenously, LNPs distribute systemically and cells throughout the body — including cardiac tissue — produce spike protein for up to 72 hours, creating a sustained inflammatory stimulus
Protein vaccine: if accidentally IV injected, you get a fixed bolus that immediately begins degrading — no ongoing production, self-limiting exposure

The double hit problem
mRNA vaccines have two independent inflammatory sources: the LNP/mRNA delivery mechanism itself, plus the immune response to the expressed protein. Protein vaccines with adjuvants deliberately add back one inflammatory stimulus, but it's more controlled than systemic LNP distribution.

What the data shows
Mouse studies confirm IV mRNA injection causes myocarditis. Some countries (Denmark, Hong Kong) adopted aspiration before injection as a precaution. But the natural experiment — comparing myocarditis rates between countries that did and didn't adopt aspiration — has not been published, leaving your core hypothesis scientifically open.

What Comes Next

The platform's key advantage is speed. Once a protein target is identified, the mRNA sequence can be designed computationally and synthesised in weeks. This compresses development timelines that previously took years. During COVID-19, the Moderna vaccine went from sequence design to first human dose in 63 days.

The bottleneck is now delivery. Current LNPs distribute broadly rather than to specific cell types, and tend to accumulate in the liver by default. For immune-based cancer vaccines, the goal is not to reach tumour cells directly — the immune system handles that — but to deliver mRNA efficiently to antigen-presenting cells like dendritic cells, which then orchestrate the immune response. For direct therapeutic approaches, such as gene editing or reprogramming cells within the tumour microenvironment, precise delivery to specific cell types does matter. More broadly, targeting mRNA to tissues like lung epithelium or particular immune cell populations would dramatically expand what is possible and reduce off-target effects. This is an active area of research.

mRNA vaccines are not a narrow tool for infectious disease. They are a general platform for programming cellular protein production. The COVID vaccines were the proof of concept at scale. Personalised cancer treatment may be the first demonstration of their full potential.

References

Colds, Immune System & Vaccination

Sean Pedersen — Fri, 03 Apr 2026 00:00:00 GMT

You've been catching colds your whole life. But how much do you really know about what's happening inside you — and what you can actually do about it?

What a cold actually is

A "cold" isn't a single illness — it's a loose term for upper respiratory infections caused by over 200 different viruses. Rhinoviruses are responsible for roughly half of them. The virus doesn't make you feel awful; your immune system does. The runny nose, fatigue, and sore throat are side effects of your body mounting a full-scale defence.

Colds are by definition viral — but bacteria can cause nearly identical symptoms, which is why the line blurs in practice. More importantly, a viral cold often weakens local defences (damaged mucosa, disrupted nasal cilia, inflamed tissue), letting bacteria opportunistically move in. This is how a cold "turns into" a sinus infection, ear infection, or bacterial bronchitis. The cold was viral; the complication is bacterial — and that's the one scenario where antibiotics become relevant.

A few signs the infection may have gone bacterial: symptoms that improve then suddenly worsen, a high fever above 39°C, or symptoms that show no improvement after 10–14 days. Worth seeing a doctor in any of those cases.

Surprising fact: Cold air doesn't cause colds directly — but it does dry out nasal membranes (weakening a key barrier), helps viruses survive longer in the air, and people spend more time indoors sharing recirculated air. It's the combination that drives winter colds seasonality, not any single factor.

Viruses spread mainly through two routes: respiratory droplets in the air, and contact with contaminated surfaces followed by touching your face. The average person touches their face around 20 times per hour, often without noticing.

The immune system, briefly

Think of immunity in two layers. The innate immune system is your instant first responder — it detects something foreign and triggers inflammation fast. The adaptive immune system is slower but smarter: it builds a bespoke response to a specific threat, then keeps a memory of it. That memory is why you rarely catch the same strain of flu twice.

When a vaccine introduces a fragment of a pathogen (or instructions for your cells to make one), it triggers the adaptive system to build that memory — without you ever getting sick from the real thing.

Mechanism worth knowing: mRNA vaccines work by mimicking how viruses hijack your cells. The vaccine delivers synthetic mRNA into your cells — just like a virus injects its own genetic instructions. Your ribosomes read that mRNA and manufacture the target protein (the spike protein). Your immune system detects this foreign protein, mounts a response, and builds antibodies and memory cells — training itself without you ever being exposed to the actual virus.

Hygiene habits that actually work

Most people wash their hands, but the details matter far more than the act itself.

20 seconds, not 5. The friction and duration matter as much as the soap. Hum "Happy Birthday", "Praise the Lord" or "Hail Satan" twice as a rough timer.
Water temperature is irrelevant. Hot vs. cold makes no practical difference to pathogen removal. Soap and friction do the work.
Masks filter outbound too. Wearing a mask when you're mildly ill reduces what you breathe onto others — regardless of their vaccination status.
Ventilate rooms. A brief burst of fresh air dramatically drops indoor viral load. Even 5 minutes with a window open helps.
Sleep is immune support. People sleeping under 6 hours are over 4x more likely to catch a cold when directly exposed to a rhinovirus.
Vitamin C won't prevent colds. High-dose vitamin C doesn't stop you getting sick. It may cut duration by about half a day in some studies — modest at best.

Things that may surprise you

Antibiotics do nothing for colds — they target bacteria, not viruses. Taking them unnecessarily accelerates antibiotic resistance and damages your gut flora unnecessarily.
Herd immunity isn't all-or-nothing. Every additional vaccinated person reduces transmission pathways, protecting those who medically can't be vaccinated.
You become contagious with a cold 1–2 days before you feel any symptoms — meaning you're spreading it before you even know you have it.
The gut hosts roughly 70% of your immune cells. What you eat genuinely affects how well your immune system can respond to threats.
Fever is a feature, not a bug. Elevated body temperature actively slows viral replication and accelerates immune cell activity.
Some people carry cold viruses and never get symptoms — they're infected but their immune response stays subclinical.

The practical bottom line

The boring interventions remain the best ones: wash hands properly and often, sleep consistently, stay up to date with vaccines, and stay home when you're contagious. The immune system is sophisticated enough to do most of the work — your job is mostly to give it the conditions it needs, and not to undermine it.

CBSNEWS: Tic-Tac UFO Article

UFO / UAP Phenomenon

Sean Pedersen — Wed, 01 Apr 2026 00:00:00 GMT

Credible UFO / UAP reports date back to WWII, when Allied pilots over both the Pacific and European theaters encountered unexplained aerial objects they called "Foo Fighters." The modern era began in June 1947 with Kenneth Arnold's sighting near Mount Rainier — prompting the U.S. Army Air Force to conclude the phenomenon was real and launch its first official investigation, Project Sign.

The debate shifted fundamentally in 2017 when the Pentagon began declassifying sensor footage and credentialed government insiders, started making specific public claims about what the U.S. government knows and may be concealing.

Documenteries like the "The Age of Disclosure" brought to a wider audience the narrative that US government officials say it is real and a real threat to US national security.

Historical Sightings

While the modern era of credible military reporting began with the WWII Foo Fighters, civilians and chroniclers recorded strange aerial events centuries earlier. Two regions left especially detailed records: Reformation era Germany, with printed broadsheets and woodcuts, and China, where careful scholars also filed reports.

Nuremberg, 14 April 1561

On the morning of 14 April 1561, citizens of Nuremberg reported a mass aerial event at sunrise lasting roughly an hour. A broadsheet printed by Hans Glaser later that month described "globes," "crosses," and cylindrical objects in red, blue, and black moving across the sky. The pamphlet claims the shapes appeared to fight one another, then fell to earth where smoke rose at the impact points. The woodcut is one of the earliest printed records of a UAP style event in Europe and was later popularized in Carl Jung's 1958 book on flying saucers.

Basel, 1566

Five years later and a few hundred kilometers south, witnesses in Basel reported similar phenomena on 27 and 28 July and again on 7 August. The historian Samuel Coccius described "large black spheres" moving with great speed in front of the sun, with some appearing to burn red before vanishing. Basel sits in modern Switzerland, but in 1566 it belonged to the German speaking cultural sphere of the Holy Roman Empire, and the report uses the same illustrated pamphlet format common across Reformation era central Europe.

Yangzhou, around 1080

A century before printing reached Europe, the Chinese polymath Shen Kuo wrote about a recurring aerial object in his "Dream Pool Essays" (1088). Locals in Fanliang, near Yangzhou, said an object "as bright as a pearl" appeared over a lake. Shen reported that the object opened to emit a great light, "like the rising Sun, lighting up the distant sky and woods in red." Witnesses nicknamed it "the Pearl," and a viewing platform called the Pearl Pavilion was built so travelers could watch for it. Shen Kuo is also credited with the first known description of the magnetic compass and a remarkably modern theory of erosion, which makes his account unusually careful for the period.

Recent Notable Incidents

Nimitz 2004 incident (Tic-Tac UFO)

Multiple military witnesses (David Fravor, Alex Dietrich, radar operators on USS Princeton).
Separate sensor tracks (F/A-18’s APG-73 radar, AN/SPY-1 ship radar, FLIR video).
Chain-of-custody FLIR footage officially released by the Pentagon.

Hangzhou, 7 July 2010

Modern China has a striking civilian aviation case as well. On the evening of 7 July 2010, Xiaoshan International Airport near Hangzhou shut down for about four hours after a flight crew on approach reported an unidentified object in its airspace. Eighteen flights were diverted or delayed. Photographs taken from the ground showed a long glowing trail. Chinese authorities never released a clear official explanation, and the case remains one of the more widely reported civilian airport UAP events of recent decades.

ChinaDaily: UFO remains a mystery

Pentagon FLIR videos

Declassified and published “GIMBAL”, “GOFAST”, “FLIR1” videos by the U.S. Navy/DoD in 2017-2021.
Accompanied by incident reports and pilot voice recordings.

The Age of Disclosure - Protagonists

Lue Elizondo — Former Army intelligence officer, ran AATIP for the US government. Claims: UAPs are real, non-human, and represent a national security threat; a secret "Legacy Program" has been retrieving and reverse-engineering non-human craft since 1947; a disinformation campaign was run against him by the Pentagon; he and David Grusch were allegedly threatened with assassination.

Jay Stratton — 16-year senior intelligence official on UAP, first director of the UAP Task Force. Claims: The Legacy Program withheld from Congress, the SecDef, and the President; non-human bodies were recovered in crash retrievals; religious fundamentalists inside the Pentagon actively blocked UAP investigation.

Christopher Mellon — Former Deputy Assistant Secretary of Defense for Intelligence. Claims: UAP is a real, ongoing phenomenon ignored by senior leadership; he helped craft UAP legislation with Rubio; defense contractors now hold recovered technology outside congressional oversight.

David Grusch — Former NRO/UAP Task Force officer, whistleblower. Testified under oath: the US runs a multi-decade crash retrieval and reverse-engineering program; non-human biologics were recovered; people have been harmed or killed to protect the secret; funds were misappropriated to hide the program.

David Fravor — Navy Commander, USS Nimitz pilot (2004). Claims: Witnessed the "Tic Tac" UAP traveling ~32,000 mph, defying known physics; the US has nothing remotely comparable.

Alex Dietrich — Navy pilot, flew with Fravor during the Tic Tac encounter. Claims: The object defied laws of physics and gravity in ways no conventional craft could.

Ryan Graves — Navy pilot. Claims: UAPs were seen nearly daily around Navy operations; a black cube inside a clear sphere passed between two aircraft at close range.

Garry Nolan — Stanford pathology professor. Claims: Military personnel who encountered UAPs suffered serious biological harm including brain scarring and a ~25% mortality rate within 7 years; he analyzed their medical data for the CIA and an aerospace company.

Hal Puthoff — Physicist. Claims: UAPs likely use a warp bubble / space-time engineering principle, explaining all five observables (hypersonic speed, instant acceleration, low observability, transmedium travel, anti-gravity); energy requirements exceed 1,100 billion watts.

Brett Feddersen — Former NSC aviation security director / FAA National Security Programs. Claims: The FAA does not track UAPs, putting aviation safety at risk; the US is "behind the power curve" on this technology.

Various nuclear weapons witnesses — Multiple Air Force and missile silo personnel claim UAPs have both deactivated and activated US nuclear missiles, and hovered silently over launch facilities.

Although not part of the documentary the story of Bob Lazar, seems to fit the narrative of the show.

The 4 Hypotheses for UAP

Foreign adversary technology: China, Russia, or another human group (some even speculate based on secret german nazi research on new types of anti-gravity propulsion systems) has developed craft so advanced it constitutes the biggest intelligence failure in modern American history.
US government counter-intelligence cover: UAPs are a cover story concealing a highly classified US program that has drifted outside congressional oversight.
Extraterrestrial or interdimensional origin: The craft originate from another civilization, dimension, or possibly even Earth itself (cryptoterrestrial, time travelers, or an ancient hidden species).
A combination of the above three: Some UAP activity may be foreign reverse-engineering of retrieved non-human craft, some may be US black programs, and some may be genuinely non-human in origin.

Conclusion

The UFO / UAP phenomenon has moved into mainstream focus and warrants more research and funding. Current stigma (negativ cracknut sentiment) needs to be overcome. Current publicly available data (videos) point to advanced technology existing capable of near-instant acceleration. And multiple high steak US gov. witnesses claiming the existence of such aircraft and even non-human bodies - until these aircraft or bodies are accessible to the public other explanations remain possible though.

References

How AI and mRNA vaccine helped a dog with cancer

Sean Pedersen — Tue, 17 Mar 2026 00:00:00 GMT

A Sydney tech entrepreneur, Paul Conyngham, had a rescue dog, Rosie, with advanced mast cell cancer and months to live after standard treatment.

He paid for Rosie's tumor sequencing through the University of New South Wales (UNSW) and then used ChatGPT and AlphaFold to help analyze mutations and think through possible targets and treatment options.

Conyngham is an experienced machine-learning practitioner, not a complete novice to technical fields, and he collaborated with scientists, including Prof. Pall Thordarson (X Profile) at UNSW’s RNA Institute.

Prof. Thordarson and his team used the data to design a personalized mRNA cancer vaccine for Rosie in under two months, describing it as the first time such a bespoke vaccine had been designed for a dog at their institute.

Rosie received injections starting in December 2025; within weeks the main tumor shrank dramatically (around 75% in one report), her energy and coat improved, and she is reported as alive and more active, though not definitively “cured.” Prof. Thorardson and others are still actively involved in curing Rosie.

It took about three months and extensive paperwork to obtain ethics approval for a veterinary trial for Rosie.

References

X Thread by Prof. Thordarson
Relevant X posts by Paul: Day #1, #2, #3, #4, #5, Images, DNA sequencing,
Fortune article
India Today tech article
NDTV article

#idea

The XZ Backdoor: A Near-Catastrophic Linux Hack

Sean Pedersen — Tue, 17 Mar 2026 00:00:00 GMT

A sophisticated 2024 supply-chain attack nearly gave an unknown hacker (group) access to millions of internet servers.

The Setup
Linux runs the world: servers, supercomputers, Android, nuclear submarines. Its security model relies on open source code being widely scrutinized ("Linus's Law"). But the ecosystem depends on thousands of tiny projects, often maintained by a single unpaid volunteer.

The Target
Lasse Collin, a Finnish developer, had maintained XZ (a widely-used lossless compression tool) alone for 20 years. Burned out and struggling with mental health, he was vulnerable to offers of help.

The Attack
A persona called Jia Tan spent ~2.5 years:

Building trust as a helpful contributor to XZ
Taking over as maintainer
Secretly embedding a backdoor inside binary test blobs (invisible in normal code review)

The backdoor exploited a precise chain: XZ (liblzma) → dependency of OpenSSH → hijacking the RSA authentication step via IFUNC resolvers and dynamic audit hooks, creating a master key to any server running the infected version.

The Near-Miss
Microsoft engineer and PostgreSQL developer Andres Freund noticed a tiny ~400ms SSH slowdown while testing an unstable Debian release - not even looking for security bugs. He kept investigating and discovered the backdoor in March 2024, just weeks before it would have shipped in major enterprise Linux releases (RHEL, Fedora, Ubuntu).

Who Was Jia Tan?
Almost certainly a nation-state actor - the operation was too patient and expensive for criminals. Timestamping and other breadcrumbs can be manipulated and thus provide no hard evidence, so no one knows for certain. Jia Tan vanished immediately after discovery.

The Bigger Lesson
The real vulnerability wasn't the code - it was the structure of the project. Critical infrastructure was resting on one exhausted, unpaid volunteer. This was not the failure of the maintainer but a structural issue of how we build critical open-source software.

To future proof open-source software, we need to identify other high-impact software projects with similar vulnerable structure (few maintainers) and pour more resources into them. And also increase automated security tests for all dependencies of high-impact projects like OpenSSH using static analysis and automated coding agents (LLM).

Similar future attacks maybe even more sophisticated by taking legitimate control of criticial open-source software for years - building up trust by years of reliable and helpful work - and when needed dropping a malicious version. A more resilient structure for critical open-source projects would be decentral and select a group of verifiers randomly from a group.

References

Veritasium YouTube Video

Urban Living

Sean Pedersen — Sun, 01 Mar 2026 00:00:00 GMT

Living in a metropol (big city) is a paradoxical thing: so many humans packed into a tight place but many feel lonely and disconnected - in contrast with living in a small community where social bonds are much stronger.

Naturalize the city landscape - the whole city should look and feel like a beautiful garden park. Bring back beautiful architecture that integrates art works of local craftmen. Use animal friendly architecture everywhere (spotted glasses to prevent flying bird deaths, small water body exit bridges to prevent animals from drowning, etc).

Create communal rooms in apartment building stairwells for neighbors to connect and chat over tea or coffee and to swap items.

AI Safety Farce

Sean Pedersen — Sun, 01 Mar 2026 00:00:00 GMT

Major AI (LLM) companies like Anthropic and OpenAI pride themselves by investing into AI safety research (which they mainly define as AI alignment aka prevent the agent from going rogue) - but they keep a blind eye on safely deploying AI for society as whole, by not investing heavily into private LLM inference, which would make LLMs private and secure (in the interest of the user) and would prevent providers from collecting user data (which is not the interest of these companies investors).

Instead they are building the most sophisticated mass digital surveillance machine the world has ever seen. They are normalizing a future where we share every intimate detail of our lives with their free or cheap chatbot service and we let these chatbots not only monitor everything about us but even worse we let us manipulate by them (by consuming their responses / content).

If these companies were truly interested in developing safe AI for humanity they would not build this centralized LLM powered mass manipulation system but instead invest heavily into safe AI deployment via decentral and private LLM inference (by publishing open-source models and using on-device inference or homomorphic encryption). Only this way can AI be deployed for humanity without enabling mass surveillance and manipulation.

It boils down to:
AI alignment without decentralization still concentrates power.
Concentrated AI power is a societal risk.
Therefore AI deployment architecture is as important as AI model alignment.

Open-source AI must win.

References:

#AI #privacy

Optimal Coding Agent

Sean Pedersen — Thu, 26 Feb 2026 00:00:00 GMT

My blueprint for the optimal coding agent (team).
Keeping LLMs in the smart zone using short and relevant context.

Usage patterns:

quick (small) edit
- skip to instructor agent
major edit
- full coding agent orchestra
new project
- deep research for best tech stack
- use proven project code template / snippets

Every code repo should have a SPEC.md file which details in natural language the purpose and capabilities of the project. The code base is then derived from it using coding agents. As coding agents improve in capability the code base will match the specification file more closely, more efficiently and with less bugs.

UX:

Provide idea -> discuss
Refine plan with designer -> submit to general
General creates the task tree -> review and submit
Watch the executors work -> review finished idea

IDEATOR / MUSE

Agent helps you crystallize your vision and define a hard success criterion (agent has web access to research ideas).

DESIGNER

Uses knowledge base: tool to query existing solved solutions + web access (query tools like deepwiki.com). Designer also needs relevant context to create good architecture (blog posts, papers, books, etc)

translates idea into objects / modules and interactions
defines interfaces / boundaries for modules
produces a design with specifications
discuss best tools (tech stack) for the design specs

GENERAL

Creates the battle plan - brief the troops.

ORCHESTRATOR

Generate the task tree from design + tech stack.

create modular plan with task execution dependency tree
- every task has:
  - ID
  - Problem: in- / output typed interface + internal behavior / structure
  - Criterion: success metric / test
  - Dependencies: task IDs
  - Complexity: low, mid, high
    - maps to small / big LLM
    - can assign multiple models to work on same task with different approaches
  - Context: added by INSTRUCTOR
  - State: defined, open, busy, trial, done, fail
  - StateHistory: List<(state, date, reason, author_agent)>

Refs:

https://github.com/steveyegge/beads
https://github.com/joshuadavidthomas/opencode-beads
https://github.com/Dicklesworthstone/beads_viewer
Why decentral task orchestration wins
Claude Code released tasks feature (similar to beads)

INSTRUCTOR

Provides every task with perfect (relevant) Context.

NEW: code templates for fresh projects
adds only relevant tools (MCP / CLI) to task context (prevents context bloat)
relevant and version matching documentation for coding tasks
- Coding guidelines based on programming languages / code files involved for the task
- General use: official matching version docs / blog posts
- Existing code: analyse package dependencies
  - exact (version) matching code documentation
  - source code -> extract public function interfaces with doc strings
    - Python: .venv/lib/python3.11/site-packges/$package_name
    - JS: node_modules/

Refs:

EXECUTORS

Coding agents working on tasks in parallel - working off the task tree backwards recursively (breadth first, towards the root task - the IDEA).

CREATOR

use LSP server for precise edits based on TASK problem and context
- optimal code exploration
- optimal code editing
  - use a database of code snippets / functions to save output tokens -> just query with a function signature + natural language description for existing code (ref)

VALIDATOR

validates if CREATOR solved tasks Criterion (test / metric)
- yes: task done
- no: retry task with modified context (add failures / learnings)

REFLECTOR

uses REPL / MCP to inspect live vars of code produced by CREATOR
fixes any issues spotted by VALIDATOR
persist learnings in knowledge base for future reference

TASK Life Cycle

State: defined, open, busy, trial, done, fail

defined: ID, Problem, Criterion, Complexity
open: added Context by INSTRUCTOR
busy: CREATOR is coding
trial: CREATOR finished
VALIDATOR is happy -> done
VALIDATOR is unhappy -> open
VALIDATOR gives up -> fail

Typical Project:

task 0 (root task) is the IDEA itself and is validated at last when all subtasks are done
the first task to tackle is the last task we planned out:
setup coding environment for new project

User Interface:

saves chats as text files (easy to inspect whole context)
displays (token usage/context limit) in current session

GUI UX

Spawn a webserver that shows current task tree + progress (indicate status for every task). Lets me also viel taks details (context, selected model etc).

More Ideas

A coding agent could massively save output tokens by outputting compact references to reusable code templates instead of writing everything token by token (and the code registry DB could be optimized by the whole community).
A tool would then expand those references into full boilerplate, functions, classes, tests, configs, or scaffolding. The best version is a typed, versioned, parameterized template/macro registry database.
The LLM becomes the planner and selector, while tools handle expansion, insertion, formatting, and validation.

new:
when agent sees multiple solutions explore in parallel

Minimal additions that would move it closer to “optimal”:
Add a GATEKEEPER (cheap, fast) before GENERAL
Input: idea + rough scope
Output: route → instructor-only | partial orchestra | full orchestra

Add post‑mortem synthesis (lightweight)
After task 0 validation:

what task boundaries were wrong?
what context was unnecessary?
what should become a template?
These don’t add conceptual weight—they reduce entropy over time.

References

Update: latest feature workflows of claude code is similar - it adds a workflow language that allows to explicitely plan parallel execution flows of agents.

Existing Agents:

https://www.mihaileric.com/The-Emperor-Has-No-Clothes/
- https://news.ycombinator.com/item?id=46545620
https://pi.dev/
https://github.com/openai/codex
https://github.com/jacobsparts/agentlib - really cool project - drops the agent directly into Python repl
https://agent-flywheel.com/flywheel
Verified Spec-Driven Development
replace claude code with cheaper + faster alternatives

Eval extending existing open-source coding agents TUI or GUI apps:

https://github.com/wandb/catnip
https://github.com/block/goose seems to be really close to my vision - check it out
https://thebob.dev/ai/tools/productivity/2025/10/31/why-we-built-claude-os-and-what-it-actually-is/
https://anandchowdhary.com/blog/2025/running-claude-code-in-a-loop
https://ampcode.com/

Fast LLMs:
groq.com, https://www.inceptionlabs.ai/, https://chat.z.ai/, https://huggingface.co/chat/

#coding #AI

Decentral Software

Sean Pedersen — Wed, 25 Feb 2026 00:00:00 GMT

We need decentral software to live in peace and freedom. Decentral systems are immune to censorship, harder to manipulate by single actors and work offline.

Projects like Bitcoin and IPFS are first steps into a better future.

We need cryptographic user identities to verify it is us and we want to own our data (no more locking up data in foreign servers).

Marko Polo: broadcast your content and collab on projects P2P.

Interplanetary Memory System : use file embeddings to discover content instead of brittle crypto hashes (IPMS is a decentral search engine - Anyone can run a node and contribute content. Embeddings are verified by peers.)

Interplanetary Trust System: Rate and discover services based on your local peers you trust -> learn which services your peers (and peers of peers) praise or dislike.

Interplanetary Messaging System: A decentral, secure messenger that works.

References

Future

Sean Pedersen — Wed, 25 Feb 2026 00:00:00 GMT

Exciting times lay ahead. Some predictiions and perspectives of mine.

Certain Developments

AI will continue to reshape how we work, increasing leverage of the (creative) individual.

computer goes faster -> LLM become cheaper and faster

everyone will have LLM powered personal agent -> value lies in providing connectors to services (websites) for the agents to use (WebMCP)

Software: will be solved for all verifiable domains -> taste and data rule after
Image + Video: Image / Video / TV show production will be solved. -> again value lies in taste.

Augmented Reality glasses will gain more and more adoption -> enabling seemless digitalization.

robots: rate of progress unknown -> but certain will be huge impact on economy and geo-politics (warfare)

Space Exploration

companies (rich individuals) will continue to push into space -> building new nation like entities

Retrieval Augmented Generation

Sean Pedersen — Wed, 25 Feb 2026 00:00:00 GMT

RAG works by augmenting the context window of an LLM with text relevant to the user's query to provide a useful response (reduce hallucinations). Thus if one wants to master RAG, one has to master search.

RAG Approaches

Simple RAG: basic keyword search / semantic search
Complex RAG: advanced chunking (e.g. late-interaction / enrich with LLM -> infer questions a chunk can answer or summarize a long document) semantic search
Graph RAG: construct knowledge graph to uncover hidden (multi-step) connections for complex queries (how did x evolve over time? how does y compare to z? explain u in terms of v, etc.)
Agentic RAG:
- let LLM create search queries based on user query
- let LLM make final selection from result candidates

Granularity Problem

Different queries require different granularity (number of chunks provided as context) -> who is the CEO of Apple vs summarize the bible

Solution:
relevant segment extraction (RSE) - https://d-star.ai/dynamic-retrieval-granularity

just use paragraph-sized chunks (~200 tokens), prepend global document context (title, section) -> embed and store position in document (line number / page position)

embed query -> find nearest chunks -> compute continuous chunk segments in documents

Hallucination Problem

How to reduce hallucinations? How to increase the response rate of "I do not know based on available information" -> search needs to be capable to indicate high uncertainty / unrelated results only -> leading to I do not know response

Fact check using semantic search and fine-tuned LLM: https://github.com/stanford-oval/WikiChat

RAG Pipeline

Based on user input query -> use LLM to generate relevant search queries -> use semantic search to find cadidates (below a similarity threshold) -> Reranker to filter (fast) -> use LLM (slow) to filter for relevancy -> use filtered results to generate final answer

Key Points

agentic search - agent decides how to search: hybrid search (semantic + keyword), custom SQL query, etc.
- LLM should transform user query before search, as user query will not always be the right format to answer the question it entails. f.e. what are the trends in tech? -> is very abstract, LLM can transform it into a format that will match the answer document semantics
query expansion
embedding chunking strategy
graph-enhanced retrieval (supermemory etc.)
domain specific benchmark (use powerful commercial LLM's to eval answers of local LLM's) to work data driven

UX:

show progress (searching / filtering / analyzing)
show user citations (used results)

RAG vs Fine-Tuning

When to pick what:

Dimension	RAG	Fine-Tuning
Corpus volatility	Fast changing (docs added daily)	Static / slow changing
Knowledge type	Factual lookup, citations needed	Style, tone, format, reasoning patterns
Traceability	Sources are visible to user	Black-box, hard to attribute
Update cost	Re-index (cheap, minutes)	Re-train (expensive, hours-days)
Hallucination risk	Lower (grounded in retrieved text)	Higher (memorized, can drift)
Cold start	Works with zero training data	Needs labeled examples
Latency	Higher (search + LLM)	Lower (single forward pass)
Best for	Q&A over private docs, support, news	Domain language, niche jargon, structured output

Use RAG when:

corpus changes faster than you can retrain
citations / source attribution are required (legal, medical, support)
knowledge is too large to fit in weights or context
you need to gate access (per-user permissions on documents)

Use Fine-Tuning when:

corpus is small, stable and dense with niche language (medical codes, legal jargon, internal product names)
you need a specific output format / style the base model fails at
latency budget is tight (no time for retrieval)
task is reasoning-heavy and pattern-based, not fact-based

Hybrid (usually the right answer for production):

fine-tune embedding model on domain pairs to improve retrieval quality
fine-tune the generator LLM for output format + citation behavior
keep RAG for the actual facts -> get the best of both
https://x.com/_avichawla/status/1964939267823775997

Local RAG

Use dspy to optimise system prompts.

Existing Solutions:

https://github.com/Cinnamon/kotaemon: Chat with PDFs
https://github.com/weaviate/Verba: Chat with PDFs
https://www.onyx.app/: Integrated solution connecting to many clouds
https://github.com/stanford-oval/WikiChat: Minimal hallucinations
https://github.com/D-Star-AI/dsRAG: Advanced chunking for text
https://github.com/weaviate/elysia: Agentic RAG (can create custom DB queries + semantic search etc.)
https://github.com/HKUDS/LightRAG
https://github.com/infiniflow/ragflow
https://github.com/langgenius/dify
https://github.com/datastax/ragstack-ai
https://github.com/neuml/txtai
Perplexica
Eliminates RAG: https://github.com/microsoft/KBLaM

Retrieval:

Memory Systems:

Benchmarks

You can not improve, what you can not see - running evals for your RAG apps is crucial to ship useful products (bad example: Elysium by weaviate).

Eval the pipeline in stages

A RAG system has two failure modes that compound: bad retrieval and bad generation. Eval them separately or you will not know what to fix.

1. Retrieval metrics (does the right chunk make it into context?)

Recall@k: out of the top-k retrieved chunks, did at least one ground-truth chunk appear? (most important - if the answer is not in context, generation can not save you)
MRR (Mean Reciprocal Rank): 1 / rank of first relevant chunk - rewards putting the right chunk near the top
nDCG@k: rewards ranking quality when there are multiple relevant chunks of varying importance
Context precision: fraction of retrieved chunks that are actually relevant - low precision wastes tokens and confuses the LLM

2. Generation metrics (given good context, is the answer good?)

Faithfulness / groundedness: every claim in the answer is supported by the retrieved context (catches hallucinations)
Answer relevancy: answer addresses the actual question, not a tangent
Citation accuracy: do the cited chunks actually contain the cited claim?
Answer correctness: matches a gold-standard answer (semantic similarity, not exact match)

3. End-to-end metrics

Task success rate: did the user get what they needed? (the only metric that really matters)
"I do not know" rate: how often does the system correctly refuse vs hallucinate when no answer exists - test with adversarial out-of-corpus queries
Latency p50 / p95: real users care
Cost per query: tokens in + tokens out + embedding + reranker calls

Building a domain benchmark

Generic benchmarks (BEIR, MTEB) tell you about general retrieval quality. They do not predict performance on your private corpus. Always build a domain eval set:

Collect 50-200 real queries from logs, support tickets, or user interviews - synthetic queries miss the long tail of how users actually phrase things
Generate ground truth - for each query annotate (a) which chunks contain the answer (b) the gold answer text. Use a strong LLM (Claude Opus, GPT-5) to draft, human to verify
Cover edge cases - multi-hop questions, queries with no answer in corpus, ambiguous queries, queries that need recent docs
Freeze the set - if you keep changing it you can not compare runs over time
Track regressions per query - aggregate scores hide that v2 fixed 5 queries but broke 3 others

LLM-as-judge

For groundedness, relevancy, correctness - use a strong LLM to score outputs of your (cheaper, smaller) production LLM. Cheap, scales, correlates with human judgment when prompted well.

Caveats:

judge model bias (LLMs prefer their own family's outputs)
ask for binary judgments + reason, not 1-10 scores (more reliable)
spot-check with humans on 10-20% of cases to keep the judge honest

Tools

FinanceBench - complex multi-hop finance analysis
EnterpriseRAG-Bench - simulates a company
KITE - retrieval benchmark
Ragas - faithfulness, answer relevancy, context precision/recall out of the box
TruLens - feedback functions and tracing
DeepEval - pytest-style RAG evals
promptfoo - eval harness with regression tracking
run evals in CI - block PRs that regress recall@k or faithfulness beyond a threshold

References

#ML #coding

Messenger

Sean Pedersen — Wed, 25 Feb 2026 00:00:00 GMT

Humanity is in need of a fun to use, secure and decentral messenger.

User identity can be tied to email (for account recovery)
Decentralized (device to device works) + federated (optional servers help make the network go fast)
E2EE message content (msg, sticker, etc.) + metadata (who msgs whom)
Connect (share keys) by scanning QR Code - “Invite link” fallback
Thick client (user keeps message history - default is 2 months) + thin server (just relay messages)
Use Rust / Erlang for server
Default: use internet, fallback: Wi-Fi Mesh; Bluetooth (like bitchat)

Chat Features (Telegram has best chat UX):

group chats
file transfer
stickers

Account:

public key
username
verified contacts

Matrix drawbacks:

Olm/Megolm does not offer forward secrecy for group messaging
Olm/Megolm does ensure end-to-end encryption for message data, but not for metadata.
Federation makes it challenging to be GDPR compliant
Synapse is very heavy, other implementations are less production ready
For better or worse, the matrix foundation is under UK jurisdiction.

Existing Chat Shitshow

Whatsapp and Telegram are security nightmares. WA being centralized and has no metadata encryption. Telegram is centralized and has no encryption by default.

Signal has best encryption but is centralized -> trust me bro mentality.
Matrix has horrible UX and no metadata encryption.
Berty chat based on IPFS https://berty.tech/messenger/
Bitchat: P2P bluetooth
Jami: P2P, https://jami.net/
https://www.mindtheclub.com/white-paper.html

MLS (secure group messaging)

MLS = a standardized group key agreement + encryption protocol
(RFC 9420, finalized 2023)

Secure group messaging with forward secrecy, post-compromise security, and efficient membership changes.

Linux Desktop

Sean Pedersen — Wed, 11 Feb 2026 00:00:00 GMT

My choice: CachyOS (Arch Linux) with GNOME DE

CachyOS is tweaked for performance, has a GUI installer + easy disk encryption setup
GNOME looks shiny and can easily be stripped down to avoid bloat

Awesome Software

File Manager:

Thunar: very lightweight 50MB RAM
Dolphin: best feature set
- supports showing dir size bytes
- supports sftp (ssh remote access)
PCManFM

Code Editor:

Zed: fast + lightweight (similar to VS Code)
Marko: WYSIWYG markdown editor

Media Player:

PW Manager:

KeePassXC

Terminal:

ghostty: fast
guake (drop down terminal)

Shell:

ZSH + ZEAL

Process Monitor:

NeoHtop / Neop (my fork)
Mission Center
Glances

Useful Daemons:

Tweaks & more:

ananicy-cpp: prevents system slow down on high disk usage (CachyOS ships with it)
OpenZFS: durable file system with useful features

Honorable Mentions

Distros:

Fedora
- Nobara (optimized for gaming / streaming)
Debian (solid)
Ubuntu Server (solid)

Desktop Envs:

KDE Plasma
HyprLand
XFCE

Dishonorable Mentions

Many GNOME apps are RAM hogs while providing little functionality, I recommend to uninstall: nautilus, gnome-software, gnome-contacts, gnome-terminal
Avoid oh-my-zsh it is bloated and slow.
Budgie DE is a buggy shit show.

TO TRY

XFCE: promising low resource DE (though no wayland support yet)
NixOS

InterPlanetary File System

Sean Pedersen — Tue, 10 Feb 2026 00:00:00 GMT

IPFS is a content based decentralized file system powered by Merkle DAG (similar to git) - allowing for local networking between nodes (separate from the internet). This technology is revolutionary and the key to a digital future with more freedom and privacy.

The IPFS protocol addresses content by what it is (a cryptographic hash - named CID), not where it is (like http), so the same content can be fetched from any peer that has it. This makes content immutable, cacheable and available offline or over local networks once retrieved.

The IPNS protocol provides a fixed IPNS name with a mutable pointer to an IPFS CID, allowing the referenced content (CID) to change over time while the IPNS name stays the same (similar to http links).

Install IPFS Desktop

IPFS Browser Extension (so ipfs:// and ipns:// work using your IPFS node):

Publishing Static Website

Configure OUT_DIR (should contain your index.html) and IPNS_KEY (an id for your website).

#!/usr/bin/env sh
set -e

OUT_DIR="out/"
IPNS_KEY="your-website-id"

# Create IPNS key if missing
if ! ipfs key list | grep -q "$IPNS_KEY"; then
  echo "🔑 Creating IPNS key: $IPNS_KEY"
  ipfs key gen "$IPNS_KEY"
fi

# Add site to IPFS
echo "📦 Adding $OUT_DIR to IPFS..."
CID=$(ipfs add -r -Q --cid-version=1 --raw-leaves "$OUT_DIR")
echo "✅ CID: $CID"
echo "🌐 Direct IPFS link (works offline):"
echo "http://ipfs.io/ipfs/$CID"

# Publish to IPNS
echo "🔗 Publishing $CID via IPNS key: $IPNS_KEY..."
ipfs name publish --key="$IPNS_KEY" /ipfs/"$CID"

# Get IPNS hash
IPNS_NAME=$(ipfs key list -l | grep "$IPNS_KEY" | awk '{print $1}')
echo "🌐 Access your blog via IPNS (stable link - works offline):"
echo "ipns://$IPNS_NAME"
echo "🌐 Access your blog via IPNS gateway (works online even without IPFS installed):"
echo "https://$IPNS_NAME.ipns.dweb.link"

Decentral IPNS Discovery

Fipsy: Decentral discovery and sharing of content using IPFS. This works in a local network - allowing true decentralized file exchange.

Show connected nodes (returns list of $NODE_ID): ipfs swarm peers
Discover public self IPNS content of node: ipfs ls /ipns/$NODE_ID
Show index of peer (only there if setup): ipfs cat /ipns/$NODE_ID/index.json

Publish all your IPNS keys as an index (for other peers in your local network to discover):

#!/usr/bin/env sh
set -e

DISCOVERY_DIR=".ipns-index"

echo "📡 Building IPNS discovery index..."

# Prepare directory
rm -rf "$DISCOVERY_DIR"
mkdir -p "$DISCOVERY_DIR"

# Generate index.json
echo "🧾 Generating index.json..."

echo '{ "ipns": {' > "$DISCOVERY_DIR/index.json"

FIRST=1
ipfs key list -l | while read -r KEY_ID KEY_NAME; do
  [ "$KEY_NAME" = "self" ] && continue

  if [ $FIRST -eq 0 ]; then
    echo ',' >> "$DISCOVERY_DIR/index.json"
  fi
  FIRST=0

  printf '  "%s": "%s"' "$KEY_NAME" "$KEY_ID" >> "$DISCOVERY_DIR/index.json"
done

echo '' >> "$DISCOVERY_DIR/index.json"
echo '} }' >> "$DISCOVERY_DIR/index.json"

# Generate index.html
echo "🌐 Generating index.html..."

cat > "$DISCOVERY_DIR/index.html" <<'EOF'



  
  IPNS Index
  


  IPNS Index
  
EOF

ipfs key list -l | while read -r KEY_ID KEY_NAME; do
  [ "$KEY_NAME" = "self" ] && continue
  echo "    $KEY_NAME $KEY_ID" >> "$DISCOVERY_DIR/index.html"
done

cat >> "$DISCOVERY_DIR/index.html" <<'EOF'
  


EOF

# Add to IPFS
echo "📦 Adding discovery index to IPFS..."
CID=$(ipfs add -r -Q --cid-version=1 --raw-leaves "$OUT_DIR")
echo "✅ CID: $CID"

# Publish under self
echo "🔗 Publishing discovery index under self..."
ipfs name publish --lifetime=1m /ipfs/"$CID"

echo "🎉 Done!"
echo
echo "🔍 Discoverable via:"
echo "  ipfs ls /ipns/$(ipfs id -f='')"
echo "  ipfs cat /ipns/$(ipfs id -f='')/index.json"
echo "  http://ipfs.io/ipns/$(ipfs id -f='')"
echo "  https://$(ipfs id -f='').ipns.dweb.link"

Kubo: Standard IPFS Implementation in Go
Helia: TypeScript IPFS Implementation for browsers
Peergos: A p2p, secure file storage, social network and application protocol
Berty: p2p, decentral, secure messenger (not yet open-sourced)
Git IPFS bridge
- https://discuss.ipfs.tech/t/ipfs-remote-bridge-for-git-version-1-0-0-is-now-available/19897
- Alternative: https://radicle.xyz
IPFS Forum
IPFS Ecosystem / IPFS Project List
Package Manager Integrations
File encryption

Commands

Initialize local node in working dir: ipfs init

Start long running node process: ipfs daemon

Add file: ipfs add $FILE_PATH

Pin file at node: ipfs pin $FILE_PATH

Show connected nodes (returns list of $NODE_ID): ipfs swarm peers

Discover public self IPNS content of node: ipfs ls /ipns/$NODE_ID

Fetch file from node: ipfs get $CID

Fetch dir from node: ipfs get -r $CID

Publish to public self IPNS: ipfs name publish /ipfs/"$CID"

Publish to IPNS key: ipfs name publish --key=" $I P N S_{K} E Y " / i p f s / "$ CID"

Update self IPNS of peer: ipfs name resolve --nocache /ipns/$NODE_ID

Resolve IPNS name to CID: ipfs resolve -r $IPNS_NAME

Publish static website: http://docs.ipfs.tech.ipns.localhost:8080/how-to/websites-on-ipfs/multipage-website

Node WebUI: http://localhost:5001/webui

Synchronize Pins across distributed daemons: https://github.com/ipfs/ipfs-cluster

References

QUESTIONS:

Can I replace SyncThing with IPFS?
Can I run an IPFS http proxy that automatically replaces HTTP requests with local IPFS response if available (basically an IPFS cache for HTTP, so stuff works offline in my local net)? Basically I want a daemon that caches HTTP/S downloads in IPFS and will use the cache for future HTTP/S requests with same URL.
Can we use IPFS to create a content-based package manager to lock down dependencies accurately?
IPFS over local bluetooth / wi-fi networks?
Can I run a website discoverable by anyone in my local network via IPFS? Yes via IPNS self - see: Publish your IPNS name keys in Self Index.
Can we host open-source code on IPFS similar to Radicle?
Why does http://ipfs.io/ipfs/ work better (offline) than ipfs:// -> ipfs.io usage does not destory relative links in websites... WTF why does ipfs:// destroy them

TODO:

[DONE] Make the IPNS self a standard to discover local IPNS content via index.html and index.json
[DONE] Create an app that helps you publish websites and browse from local peers -> Fipsy

#privacy #coding

Absurditäten unserer Gesellschaft

Sean Pedersen — Tue, 10 Feb 2026 00:00:00 GMT

Ein paar Dinge die in der BRD komplett falsch laufen aber als normal erachtet werden.

"Absolute Geldstrafen für Ordnungswidrigkeiten und Straftaten" - sind absolut ungerecht. Reiche Menschen werden nicht abgeschreckt und arme Menschen unverhältnismäßig bestraft. Nur relative Geldstrafen die sich prozentual am Einkommen des zu Bestrafenen errechnen, haben für alle Einkommensschichten den gleichen abschreckenden Effekt (s. Finnland wo es so schon gut funktioniert).
"Ungleichheit vor Gericht" - Die finanziellen Möglichkeiten eines Angeklagten haben leider einen erheblichen Einfluss auf das Urteil. Die Situation ist ähnlich wie mit privaten und gesetzlichten Krankenkassen. Wer sich mehr Zeit von erfahrenen Anwälten leisten kann, hat schlichtweg bessere Chancen vor Gericht. Eine pot. Lösung wäre ein System, das nur noch aus gesetzlichen Pflichtverteidigern besteht (Ich habe noch nicht viel über mögliche bessere Alternativen nachgedacht).
"Monopole zu dulden für kritische Infrastruktur wie öffentliche Verkehrsmittel."
- Kein Anreiz für den Monopolisten guten Service zu bieten -> kritische Infrastruktur sollte vom Staat betrieben werden und von den Bürgern durch Wahlen geformt werden.
- Es kann nicht wahr sein dass die Nutzung von öffentlichen Verkehrsmitteln Geld kostet. Sie sollten über Steuern finanziert werden und damit Schwarzfahrer wie Ticket-Kontrolleure verschwinden. Es gehen jedes Jahr Menschen ins Gefängnis weil sie ihr Ticket nicht bezahlen können - ein absolutes Unding. Ich empfinde nichts als Verachtung für jeden Menschen der sich als Kontrolleur, Polizist, Justizvollzugsbeamter oder Richter an diesem unmenschlichen System beteiligt.
"Die Kriminalisierung von Drogen." - verschwendet wertvolle Polizeiarbeit, verpestet die Umwelt, lässt potentielle Steuergelder liegen und treibt Konsumenten in echte kriminelle Kreise. Alle Drogen sollen nur in Apotheken oder extra Abgabestellen für Volljährige verfügbar sein - begleitet mit Präventionsarbeit und Hilfsangeboten für Abhängige. Alkohol und Tabak sind ebenso Drogen wie jede andere und sollten nicht in Lebensmittelgeschäften verfügbar sein.
"Toleranz von nachweislich toxischen Online-Plattformen wie TikTok, YouTube Shorts, Instagram und co." Jegliche Art von endlos Feeds in digitalen Apps gehört verboten. Tech-Unternehmen machen bereits Kinder zu abhängigen unmündigen Konsumenten. (Selbst ein Verbot bekämpft lediglich das Sympton, die Ursache ist die Normalisierung des digitalen Überwachungskapitalismus durch unstillbare Gier nach Profit und Macht.)
"Die Zweiklassengesellschaft der Privat- und Gesetzlichversicherten." Der Zugang zum Gesundheitssystem darf nicht eine Frage des Geldes sein. Schafft die Privatkasse ab.
"Die 5% Klausel für Parteien" - Der Sinn soll seien, das Parlament regierungsfähig zu halten, indem Zustände wie in der Weimarer Republik vermieden werden (zu viele Splitterparteien). Doch der wahre Effekt ist die Lähmung neuer politischer Kräfte. Neue Parteien haben kaum Sichtbarkeit und Wähler rechnen damit, dass ihre Stimme durch die 5% Klausel verpufft (was sie hemmt neue / kleine Parteien zu wählen). Eine bessere Lösung ist eine fixe Anzahl von Fraktionen im Parlament (z.B. 8) mit einer deutlich niedrigeren Parteiensperrklausel (z.b. 1%) - damit wäre das Parlament stets regierungsfähig und neue / kleine Parteien haben eine echt Chance ins Parlament einzuziehen.
"Wohnraum als Spekulationsobjekt" - Wohnraum ist ein Grundrecht, welches nicht aus Profitgier sabotiert werden darf.

Genetics

Sean Pedersen — Sun, 25 Jan 2026 00:00:00 GMT

The science of genes and their mutations in organisms.

DNA (DeoxyriboNucleic Acid) is the spiral molecular structure (double helix) made of 4 nucleotide bases (adenine, thymine, guanine, cytosine). These pair up symmetrically: A<->T, G<->C, forming a base 4 information system that encodes the instructions for living organisms to grow.

Structure of Genes

Genes are specific segments of DNA that contain instructions for producing proteins (or functional RNA). DNA molecules are packaged by coiling up into structures called chromosomes. Human cells contain 23 pairs of chromosomes - one set inherited from each parent - and each chromosome carries thousands of gene segments along its DNA sequence.

A gene is defined as a DNA segment that contains the code to produce a specific protein or functional RNA molecule. It is typically a few thousand base pairs long for humans (though can range from hundreds to a million). Genes have defined start and end boundaries identified through special combinations of base pairs (like start / stop codons).

BONUS: Take a look at the genome of a Corona virus (SARS-CoV-2)

Polymerase Chain Reaction

PCR is one of the most important breakthroughs in the last century - allowing us to read DNA samples easily by exponentially copying specific DNA sequences, making millions of copies from a tiny sample.

How it works:

Denaturation (~95°C) - Heat separates double-stranded DNA into single strands
Annealing (~55-65°C) - Short DNA primers bind to target sequences
Extension (~72°C) - DNA polymerase enzyme builds new complementary strands

This cycle repeats 25-35 times, exponentially copying the target DNA segment.

The DNA polymerase used in PCR comes from heat-loving bacteria that live in extreme environments. PCR was invented by Kary Mullis in 1983 (who had many LSD experiences).

CRISPR

CRISPR (Clustered Regularly Interspaced Short Palindromic Repeats) is a gene-editing tool that allows precise cutting and modification of DNA sequences.

How it works:

Guide RNA - Designed to match the target DNA sequence
Cas9 protein - Acts as molecular scissors, cutting DNA at the targeted location
Cell repair - The cell's natural repair mechanisms fix the break, allowing deletion, correction or insertion of genes

Where it came from: Cas9 protein originates from bacterial immune system against viruses. Bacteria store viral DNA snippets in CRISPR arrays to "remember" past infections. When the virus returns, bacteria use this memory to cut and destroy viral DNA.

Common uses:

Treating genetic diseases (sickle cell, some cancers)
Agricultural improvements (drought-resistant crops)
Research (studying gene function)

Why it's revolutionary:

Earlier gene editing was slow, expensive, and imprecise
CRISPR is fast, cheap (~$30/edit vs thousands), and highly accurate
Democratized genetic engineering

Discovered by Jennifer Doudna and Emmanuelle Charpentier, who won the 2020 Nobel Prize in Chemistry.

BONUS: You can create your custom yeast strain using this CRISPR service

Surprising Q&A

Does number of chromosome pairs correlate with complexity of an organism?

Chromosome number varies widely across species without correlation to complexity:

Humans: 23 pairs
Dogs: 39 pairs
Fruit flies: 4 pairs
Ferns: can have over 500 pairs

What matters for complexity of an organism is gene regulation, alternative splicing, non-coding RNAs and protein interactions - not just chromosome count or even total gene number.

What does it mean our DNA is 99% similar to chimps? how is DNA similarity computed?

The 99% refers to aligned sequences (comparable regions)
Doesn't account for insertions/deletions (indels) - chimps have different chromosome counts
Non-coding regions can differ more than genes
Overall genome similarity is closer to 96-98% when including structural differences

How was the tree of life calculated?

Key breakthrough:
Carl Woese (1977) used ribosomal RNA sequences to reveal three domains of life (Bacteria, Archaea, Eukarya), revolutionizing classification that was previously based on visible features.

Modern approach:

Sequences from multiple genes create more robust trees
Whole genome comparisons for fine detail
Molecular clocks estimate divergence times using mutation rates

Challenges:

Horizontal gene transfer (especially in bacteria)
Incomplete fossil record
Convergent evolution (similar traits, different origins)

How many genes mutate in human sexual reproduction on average?

In human sexual reproduction, approximately 60-100 new mutations occur per offspring on average. These are de novo mutations—changes that appear in the child but weren't present in either parent's genome. Most occur during sperm production (~80% are paternal in origin and increase with age). Most mutations are in non-coding regions and have no effect.

Geographical Genetic Heritage

Why genetic heritage services like 23andme make no sense: They claim to analyse your genome and then send you a report of your ancestry, looking something like: 10% Norh-European, 20% Russian, 70% Chinese. These numbers make no sense since they are based on DNA samples from arbitrary reference populations in these regions but humanity has been travelling earth since its inception - so any regional reference sample is just a temporal snapshot at best. To claim you are X% from region Y is thus a misleading statement. One could only say with high uncertainty: you share X% of your DNA with most humans that lived in region Y Z years ago.

Genes and Intelligence

I have often witnessed people debating how much percent of intelligence of a human is genes vs (learning) environment - 50% / 50% or 30% / 70%? This is the wrong question to ask.

The intelligence of an individual is fundamentally limited by their DNA. The intelligence potential manifested in their genes is either unlocked by positive (learning) experiences or stays just a never reached potential.

References

TODO:

Nanorobots - Precise Drug Delivery

Sean Pedersen — Wed, 07 Jan 2026 00:00:00 GMT

Medical nanorobots are tiny vehicles controlled by external electromagnetic fields, enabling them to move through patients veins. The vision is precise delivery of drugs to localized body parts like tumors to minimize side-effects of treatments like chemotherapy.

Control Mechanism

External magnetic fields manipulate nanorobots containing magnetic materials. Rotating fields create swimming motions, while gradient fields enable directional steering. Controllers adjust field strength and orientation to guide the robots through blood or tissue. Ultrasound or MRI provides real-time tracking without interfering with magnetic control.

Current Applications

Targeted drug delivery: Nanorobots carry chemotherapy drugs directly to tumors, reducing side effects from systemic treatment.

Cancer therapy: Spiky designs penetrate tumor cell membranes to improve drug absorption. Studies in mice show tumor size reduction.

Minimally invasive surgery: Microrobots navigate blood vessels to reach treatment sites without incisions.

2025 Developments

Neutrobots combine magnetic steering with chemical sensing to find targets after injection. Magnetic microrobots aggregate in aneurysm sacs using acoustic propulsion. Biodegradable designs eliminate the need for removal after treatment. Wireless magnetic communication enables coordination between multiple robots.

Technical Status

Laboratory testing demonstrates successful navigation in simulated environments and animals like mice. Materials include magnetic nanoparticles embedded in biocompatible polymers or gels. Sizes range from nanometers to millimeters depending on application. MRI-compatible tracking methods prevent image distortion during guidance.

Remaining Obstacles

Immune system recognition remains a challenge for extended circulation. Manufacturing at scale requires new production methods. Long term biocompatibility needs verification across different body environments. Regulatory approval processes for medical devices this novel are undefined.

References

Microplastics - A silent killer

Sean Pedersen — Mon, 05 Jan 2026 00:00:00 GMT

Microplastics are tiny plastic particles present in food, water, air and dust. Research links higher exposure to inflammation, fertility problems and cardiovascular issues. Complete avoidance is impossible, but intake can be reduced. Mass usage of plastics in the economy is a huge mistake (as most organisms lack metabolism to deal with it) and must be stopped.

Water Sources

Bottled water contains hundreds of thousands of microplastic particles per liter, far more than tap water. The packaging and processing add contamination. Use glass or stainless steel bottles instead.

Filter tap water with NSF/ANSI certified systems like reverse osmosis or certain pitcher filters. Boiling water before filtering may reduce particles further.

Food Storage

Never microwave food in plastic containers, even those labeled microwave safe. Heat accelerates microplastic release - avoid placing hot food in plastic or running plastic items through dishwashers.

Replace plastic cutting boards with wood or bamboo. Plastic boards shed significant particles when cut. Choose wooden spoons and metal utensils over plastic ones.

Wash rice, meat, fish, fruits and vegetables before cooking. This removes 20 to 40 percent of surface microplastics.

Food Choices

Buy fresh foods over ultra-processed items in plastic packaging. Choose loose produce when available. Reduce canned goods and plastic wrapped foods.

Limit seafood consumption, especially shellfish, which accumulates ocean microplastics. Use loose leaf tea or plastic free tea bags. Standard tea bags release billions of particles in hot water.

Household Habits

Wear natural fibers like cotton, wool, or linen. Synthetic fabrics such as polyester and nylon shed microfibers during wear and washing.

Vacuum regularly with HEPA filters. Dust and wet mop surfaces to remove settled microplastics. This reduces inhalation and ingestion from household dust.

Kitchen Utensils

Metal spoons scraping plastic containers release microplastic particles through abrasion. Plastic yogurt cups made from polypropylene or polystyrene are especially vulnerable to scratching.

Wooden or bamboo spoons are softer and cause less surface damage than metal. The best option is transferring food to glass, ceramic, or metal bowls before eating. Avoid aggressive scraping with any utensil.

Priority Actions

Start with high impact changes for maximum reduction. Stop buying bottled water. Never heat food in plastic. Use natural materials for food preparation. These steps target the main exposure pathways: water, food and air.

Minimize single use plastics like takeout containers and straws. This lowers both environmental release and personal exposure.

References

A Thousand Brains - Jeff Hawkins

Sean Pedersen — Sat, 27 Dec 2025 00:00:00 GMT

Reference Frames

Identifies so called reference frames (grids / maps) as key concept unifying all biological neural processing for robust / invariant sequence prediction (allowing sequence prediction in changing environments by using relative positions).

Key points:

Grid cells: Each cortical column uses grid cell-like mechanisms (from the entorhinal cortex) to create reference frames anchored to objects, not just spatial locations.
Object-centered coordinates: When you touch a coffee cup, each column builds a model in the cup's own reference frame - tracking features relative to the object itself, not your body or the room.
Voting mechanism: Thousands of columns process different inputs (different fingers touching, different viewpoints) in parallel. They "vote" to reach consensus on what object is present and its pose.
Location + feature: Each column stores what features exist at specific locations within an object's reference frame. Moving your sensor updates the location signal.
Compositional models: Reference frames allow hierarchical object models - a wheel has its own frame nested within a car's frame.

For abstract concepts like words in a language f.e. democracy, we create abstract maps (ref. frame) that encode how democracy is structured and how it behaves (model). This allows agents to navigate democracy: how to create a new party? etc.

Properties of reference frames:

indexable (knowing a few bits of a location, allows us to fetch the correct map & location)
structured by locality (abstract: similarity)
find routes from location / concept A to B

Useful knowledge representation uses object based models: allowing to make inferences by predicting interactions / behavior. Storing hard facts (as text) is necessary but not sufficient for a useful knowledge base. Storing knowledge in gigantic, fractured weight matrices (like used in LLMs) is useful and interesting but does not produce human like adaptive intelligence.

Language understanding entails thus creating and manipulating a small world model of the text one is reading. For example: The apple was rolling off the table. -> To understand this sentence, a model of gravity is needed to predict what will most likely happen next to the apple (hit the ground). LLMs lack this predictive word model and merely predict the next token (creating at best fractured / brittle models).

Road to AGI

Hawkins is sure that deep learning (including LLM) will not lead to AGI - as they lack model based knowledge representation. Instead he identified the following properties for realizing AGI.

Continuous learning
Learning via movement / interaction
Compositional world model: predictive (behavior) models of objects
Knowledge stored in reference frames

Critique

Hawkins maybe too focused on some specifics of the human / mammal brain structure - he identified cortical columns as the key building part of intelligence but there are other intelligent animals which lack cortical columns (smart birds like crows / ravens and octpus). A better way forward is thus to understand the structural similarities between these independently co-evolved intelligent species brains - to truly get at the essential parts for intelligence that are shared among all smart animals.

Mammals, corvids and octopi all share a columnar brain structure, while less intelligent species like goldfish and reptiles lack these and instead have nuclear neural clusters. Parrots have less pronounced columnar brain structure like corvids while showing similar cognitive abilities (but corvids seem smarter w.r.t. tool use / metacognition). Hawkins theory would be falsified by identifying a species with advanced cognition that lacks similarity to cortical columnar brain structure found in mammals.

On Page 186: Hawkins gives a false sense of evolution IMO, invoking the sense that evolution always optimizes traits of individuals but it is more random: many things stick that do not create a disadvantage to procreate instead of only traits that increase procreation chance.

References

YouTube - Jeff Hawkins NAISys: How the Brain Uses Reference Frames

#book #AI

On Intelligence - Jeff Hawkins

Sean Pedersen — Wed, 10 Dec 2025 00:00:00 GMT

Neo Cortex forms a uniform structure: 6 layers of cortical columns composed mostly of pyramidal neurons. Information flows bidirectionally: down the hierarchy to relate to known patterns and up the hierarchy to predict unknown / complete partial patterns.

Brain does not compute mathematical equations to navigate the world but creates and recalls memories. Catching a flying ball involves (muscle sequence) memories of previous catches that can be efficiently parameterized to adapt to the current flying ball situation.

Knowledge Representation (neo cortex)

sequential (easy to recite A-Z, hard to do the reverse)
invariant (to geometric translations in space and time)
auto-associative (informations connect by observing in shared context)
hierarchical (complex/abstract patterns are composed of simple patterns)
SDR: sparse distributed representations
topographic: neighboring neurons, activate to similar patterns

Learning / Forming Memories

fire together, wire together (Hebbian learning): similar patterns that repeatedly fire together, will bind together (even across different input domains), translational invariance is not only spatial but also temporal (to bind two stimuli they must not co-occur exactly at same time)

Grandmother cell hypothesis is wrong: Each neuron doesn't detect just one object. This wouldn't allow lifelong learning or resilience to cell death - instead concepts are SDRs.

#book #AI

Hyper-Primes

Sean Pedersen — Mon, 17 Nov 2025 00:00:00 GMT

A hyper-prime is a prime number p where all prime factors of p-1 and all exponents in the prime factorization of p-1 are themselves hyper-primes.

Definition

For a prime number p: find prime factorization PF(p-1):
PF(p-1) = q₁^e₁ × q₂^e₂ × ... × qₖ^eₖ
The prime p is hyper-prime if and only if:

Each prime factor qᵢ is hyper-prime
Each exponent eᵢ is hyper-prime

By definition, 0 and 1 are hyper-primes.

Table

Prime	Factors of p-1	Exponents	HyperPrime
0			True
1			True
2	1	1^0	True
3	2	2^1	True
5	2 × 2	2^2	True
7	2 × 3	2^1, 3^1	True
11	2 × 5	2^1, 5^1	True
13	2 × 2 × 3	2^2, 3^1	True
17	2 × 2 × 2 × 2	2^4	False
19	2 × 3 × 3	2^1, 3^2	True
23	2 × 11	2^1, 11^1	True
29	2 × 2 × 7	2^2, 7^1	True
31	2 × 3 × 5	2^1, 3^1, 5^1	True
37	2 × 2 × 3 × 3	2^2, 3^2	True
41	2 × 2 × 2 × 5	2^3, 5^1	True
43	2 × 3 × 7	2^1, 3^1, 7^1	True
47	2 × 23	2^1, 23^1	True
53	2 × 2 × 13	2^2, 13^1	True
59	2 × 29	2^1, 29^1	True
61	2 × 2 × 3 × 5	2^2, 3^1, 5^1	True
67	2 × 3 × 11	2^1, 3^1, 11^1	True
71	2 × 5 × 7	2^1, 5^1, 7^1	True
73	2 × 2 × 2 × 3 × 3	2^3, 3^2	True
79	2 × 3 × 13	2^1, 3^1, 13^1	True
83	2 × 41	2^1, 41^1	True
89	2 × 2 × 2 × 11	2^3, 11^1	True
97	2 × 2 × 2 × 2 × 2 × 3	2^5, 3^1	True
101	2 × 2 × 5 × 5	2^2, 5^2	True
103	2 × 3 × 17	2^1, 3^1, 17^1	False
107	2 × 53	2^1, 53^1	True
109	2 × 2 × 3 × 3 × 3	2^2, 3^3	True
113	2 × 2 × 2 × 2 × 7	2^4, 7^1	False
127	2 × 3 × 3 × 7	2^1, 3^2, 7^1	True
131	2 × 5 × 13	2^1, 5^1, 13^1	True
137	2 × 2 × 2 × 17	2^3, 17^1	False
139	2 × 3 × 23	2^1, 3^1, 23^1	True
149	2 × 2 × 37	2^2, 37^1	True
151	2 × 3 × 5 × 5	2^1, 3^1, 5^2	True
157	2 × 2 × 3 × 13	2^2, 3^1, 13^1	True
163	2 × 3 × 3 × 3 × 3	2^1, 3^4	False
167	2 × 83	2^1, 83^1	True
173	2 × 2 × 43	2^2, 43^1	True
179	2 × 89	2^1, 89^1	True
181	2 × 2 × 3 × 3 × 5	2^2, 3^2, 5^1	True
191	2 × 5 × 19	2^1, 5^1, 19^1	True
193	2 × 2 × 2 × 2 × 2 × 2 × 3	2^6, 3^1	False
197	2 × 2 × 7 × 7	2^2, 7^2	True
199	2 × 3 × 3 × 11	2^1, 3^2, 11^1	True
211	2 × 3 × 5 × 7	2^1, 3^1, 5^1, 7^1	True
223	2 × 3 × 37	2^1, 3^1, 37^1	True
227	2 × 113	2^1, 113^1	False
229	2 × 2 × 3 × 19	2^2, 3^1, 19^1	True
233	2 × 2 × 2 × 29	2^3, 29^1	True
239	2 × 7 × 17	2^1, 7^1, 17^1	False
241	2 × 2 × 2 × 2 × 3 × 5	2^4, 3^1, 5^1	False
251	2 × 5 × 5 × 5	2^1, 5^3	True
257	2 × 2 × 2 × 2 × 2 × 2 × 2 × 2	2^8	False
263	2 × 131	2^1, 131^1	True
269	2 × 2 × 67	2^2, 67^1	True
271	2 × 3 × 3 × 3 × 5	2^1, 3^3, 5^1	True
277	2 × 2 × 3 × 23	2^2, 3^1, 23^1	True
281	2 × 2 × 2 × 5 × 7	2^3, 5^1, 7^1	True
283	2 × 3 × 47	2^1, 3^1, 47^1	True
293	2 × 2 × 73	2^2, 73^1	True
307	2 × 3 × 3 × 17	2^1, 3^2, 17^1	False
311	2 × 5 × 31	2^1, 5^1, 31^1	True
313	2 × 2 × 2 × 3 × 13	2^3, 3^1, 13^1	True
317	2 × 2 × 79	2^2, 79^1	True
331	2 × 3 × 5 × 11	2^1, 3^1, 5^1, 11^1	True
337	2 × 2 × 2 × 2 × 3 × 7	2^4, 3^1, 7^1	False
347	2 × 173	2^1, 173^1	True
349	2 × 2 × 3 × 29	2^2, 3^1, 29^1	True
353	2 × 2 × 2 × 2 × 2 × 11	2^5, 11^1	True
359	2 × 179	2^1, 179^1	True
367	2 × 3 × 61	2^1, 3^1, 61^1	True
373	2 × 2 × 3 × 31	2^2, 3^1, 31^1	True
379	2 × 3 × 3 × 3 × 7	2^1, 3^3, 7^1	True
383	2 × 191	2^1, 191^1	True
389	2 × 2 × 97	2^2, 97^1	True
397	2 × 2 × 3 × 3 × 11	2^2, 3^2, 11^1	True
401	2 × 2 × 2 × 2 × 5 × 5	2^4, 5^2	False
409	2 × 2 × 2 × 3 × 17	2^3, 3^1, 17^1	False
419	2 × 11 × 19	2^1, 11^1, 19^1	True

Code

import math
from collections import Counter

def prime_factors(n):
    factors = []
    while n % 2 == 0:
        factors.append(2)
        n //= 2
    f = 3
    while f * f <= n:
        while n % f == 0:
            factors.append(f)
            n //= f
        f += 2
    if n > 1:
        factors.append(n)
    return factors

def is_prime(num):
    if num < 2:
        return False
    if num in (2, 3):
        return True
    if num % 2 == 0:
        return False
    root = int(math.sqrt(num))
    for i in range(3, root + 1, 2):
        if num % i == 0:
            return False
    return True

def is_hyper_prime(n, memo={}):
    if n in memo:
        return memo[n]
    
    # Bootstrap base cases: 0, 1, 2 are hyper-primes by definition
    if n in (0, 1, 2):
        memo[n] = True
        return True
    
    # Must be prime to be hyper-prime
    if not is_prime(n):
        memo[n] = False
        return False
    
    # Factor n-1
    factors = prime_factors(n - 1)
    factor_counts = Counter(factors)
    
    # Check all prime factors are hyper-primes
    for factor in factor_counts.keys():
        if not is_hyper_prime(factor, memo):
            memo[n] = False
            return False
    
    # Check all exponents are hyper-primes
    for exponent in factor_counts.values():
        if not is_hyper_prime(exponent, memo):
            memo[n] = False
            return False
    
    memo[n] = True
    return True

def generate_primes(limit):
    primes = []
    for p in range(2, limit + 1):
        if is_prime(p):
            primes.append(p)
    return primes

def generate_hyper_primes(limit):
    hyperprimes = []
    memo = {}
    for p in range(2, limit + 1):
        if is_hyper_prime(p, memo):
            hyperprimes.append(p)
    return hyperprimes

def generate_hyper_primes_debug(limit):
    print(f"| {'Prime':>6} | {'Factors of p-1':>25} | {'Exponents':>15} | {'HyperPrime':>10} |")
    print(f"|{'-'*8}|{'-'*27}|{'-'*17}|{'-'*12}|")
    memo = {}
    for p in range(2, limit + 1):
        if not is_prime(p):
            continue
        factors = prime_factors(p - 1)
        counts = Counter(factors)
        exps_str = ', '.join(f"{prime}^{exp}" for prime, exp in counts.items())
        hyper = is_hyper_prime(p, memo)
        factors_str = ' × '.join(map(str, factors)) if factors else '1'
        print(f"| {p:6} | {factors_str:25} | {exps_str:15} | {str(hyper):>10} |")

# Example test up to 420
generate_hyper_primes_debug(420)

References

https://en.wikipedia.org/wiki/Super-prime

#math

Elixir

Sean Pedersen — Fri, 14 Nov 2025 00:00:00 GMT

Elixir is Erlang with syntactic sugar (transpiling to Erlang code). Biggest differences to Erlang: Variables are rebindable & strings are UTF-8 encoded by default.

Parallel function process map:

defmodule Utils do
    def pmap(collection, func) do
        collection
        |> Enum.map(fn item -> Task.async(fn -> func.(item) end) end)
        |> Enum.map(fn task -> Task.await(task) end)
    end
end

Fibonacci base, memoized and tail recursion:

defmodule Fibo do
    # Exponential time complexity: O(2^n)
    # Linear space complexity due to call stack: O(n)
    def fibonacci(0), do: 0
    def fibonacci(1), do: 1
    def fibonacci(n) when n > 1 do
        fibonacci(n - 1) + fibonacci(n - 2)
    end

    # Memoized Fibonacci using caching to avoid redundant calculations
    # Time complexity reduced to O(n)
    # Space complexity O(n) for cache map
    def fibo_cached(n), do: fibo_cached(n, %{}) |> elem(0)

    defp fibo_cached(0, cache), do: {0, cache}
    defp fibo_cached(1, cache), do: {1, cache}

    defp fibo_cached(n, cache) when n > 1 do
        case Map.fetch(cache, n) do
        {:ok, result} ->
            {result, cache}
        :error ->
            {res1, cache1} = fibo_cached(n - 1, cache)
            {res2, cache2} = fibo_cached(n - 2, cache1)
            result = res1 + res2
            {result, Map.put(cache2, n, result)}
        end
    end

    # Tail recursive Fibonacci
    # Time complexity O(n)
    # Constant space complexity O(1)
    def fibo_tail(n), do: fibo_tail(n, 0, 1)
    defp fibo_tail(0, a, _b), do: a
    defp fibo_tail(n, a, b) when n > 0 do
        fibo_tail(n - 1, b, a + b)
    end
end

References

Rust

Sean Pedersen — Fri, 14 Nov 2025 00:00:00 GMT

Rust is a statically and strongly typed systems programming language that enforces memory safety (without a garbage collector) through its ownership and borrowing model, while still being efficient in the ballpark of C or C++.

In safe Rust, the compiler prevents entire classes of bugs common in C or C++, such as use-after-free and most data races. It also forces explicit handling of optional values via Option instead of allowing null references. While this shifts many potential failures from runtime to compile time (in contrast with dynamic languages like Python or unsafe manual memory management in C) - Rust does not eliminate logic errors, panics or higher-level concurrency issues like deadlocks. These properties are the reason Rust has made its way into the Linux kernel.

Rust also shines through its ergnomic package manager cargo that provides a smooth developer UX. The only real downside of Rust is its inherent complexity (slow compile speeds and huge compilation artifacts disk size) and at times verbose / ugly syntax.

Great introduction: https://fasterthanli.me/articles/a-half-hour-to-learn-rust

HTTP Server

Machine Learning

ndarray: numpy equivalent
- loading npy files (from Python numpy)
ORT: ONNX run time

GUI

Tauri: Use Rust in backend and web stack in frontend to build desktop and mobile apps
- Tauri MCP Server
GPUI: UI lib created by and used in Zed editor
- GPUI components

Text Extraction

pdf_oxide: fast PDF text extraction
ferrules: structured text extraction
https://github.com/kreuzberg-dev/kreuzberg
https://github.com/yobix-ai/extractous

Text Chunking

Concurrency

cineyma: Erlang inspired OTP-style actor framework
asupersync: Async runtime for Rust where correctness is structural: region-owned tasks, cancel-correct protocols, capability-gated effects, and deterministic replay testing

Coding Agents

OpenAI Codex
pi_agent_rust: Rust port of pi agent

References

Digital Surveillance Capitalism

Sean Pedersen — Thu, 13 Nov 2025 00:00:00 GMT

Digital sureveillance capitalism shapes how we interact with technology, the world and even how we understand its role in society. These mechanisms tend to hide the true power relations while presenting inhumane, privacy violating technological developments as neutral, natural progress.

Core Mechanisms

Solutionism

Tech firms recast hard social problems as simple tech tasks. Debate over law and justice shifts into product tweaks.

Example: using filters and bots to police hate speech instead of asking who should set speech rules. This story defends deregulation, giant scale and ignores waste, labor abuse and social harm.

Forced Transparency and Self Tracking

People are scored nonstop: credit, clicks, steps, ratings. Being watched makes us adjust to please the watcher.
Users are exposed; while platforms algorithms and business aims stay hidden. Long legal pages, secret algorithms and many confusing company layers.
Self tracking is sold as freedom; it becomes self discipline to meet outside targets: fitness badges, five‑star gig ratings. Trust is demanded; proof is withheld.

Engineered Comparison

Likes, followers, ranks, streaks keep users chasing status. Games and rewards drive time on site and data flow. This is not random manipulation; it is the profit engine. Engagement feeds ads and prediction. People stay not from weakness but because exit costs social ties.

Supporting Stories

Dataism: Numbers are treated as pure truth; method bias is ignored. Tech inevitability: "Tech will happen anyway," killing democratic choice. Disruption myth: Breaking rules is praised as innovation; law evasion is rebranded. Sharing talk: Extractive platforms pose as neutral "sharing" spaces. Security frame: More surveillance is sold as safety; emergency logic never ends.

Results

Power pools in a few firms. Risk shifts to users and workers: algorithmic gig control, data leaks, biased automation. This is masked as efficiency and empowerment. Design choices are political choices.

Human Centered Technology

...serving humans instead of exploiting them for maximum profits.

Open Source

Code you can read, verify and change. True digital independence. In the future everyone may change their apps using local LLM's.

Local First

Data lives on your devices; sync only when you choose. Works offline. Cuts surveillance. Real ownership, not unread policies.

Open Standards and Interoperability

Systems talk through public protocols. You can move and keep contacts and data. Stops lock‑in. Forces competition on quality.

Privacy by Design

Collect the least data. Encrypt end to end. Use techniques that hide individuals in stats. Make abuse hard by architecture, not fragile policy.

Democratic Governance

Co‑ops, user boards, multi‑stakeholder councils. Invite affected groups into rule making. Example: civic platforms that enable public debate and consensus.

Alternative Economic Models

Platform Co‑ops: Worker/user owned services share profit fairly; aim for good work over raw growth.
Public Digital Infrastructure: Essential digital tools should be available as public goods (just like roads) and just be maintained without interest in profiting from it.
Commons Production: Shared, open projects (Linux, Wikipedia) show large systems can run without corporate control.

Regulation

Data Minimization: Collect only what you need; delete when done. Enforce it.
Algorithm Accountability: Explanations, audits, appeal paths for decisions on jobs, housing, credit, benefits.
Antitrust: Block monopoly mergers, split dominant firms, ban self‑dealing. Separate core platform from its own services.

Digital Literacy: Teach how platforms track, rank and shape behavior. Not just code—power analysis.
Collective Action: Unions, advocacy, movements change systems; single user choices rarely do.
Cultural Norms: If people expect data rights and low surveillance, firms must adapt. Norms spread through media, art, daily talk.

Working Examples

Mastodon: Decentralized social network; users pick servers and rules. Signal: Encrypted chat funded by donations, not data extraction. Fairphone: Repairable phones built for long use. Cities like Barcelona: Public digital platforms serving citizens.

Path Forward

Change needs tools, laws, ownership shifts and new norms together. Nothing is inevitable. We can choose designs that cut surveillance, spread power, support democracy and respect dignity.

Human tech serves people, not extracts from them. It shares power, not hoards it. It grows under public oversight, not private decree. Choose and build accordingly.

References

Operational Security

Sean Pedersen — Mon, 10 Nov 2025 00:00:00 GMT

How to secure a VPS running in production (important service + customer data)

VPS Providers (EU)

hetzner: Germany
ovhcloud: France
scaleway: France

Checklist

Use ONLY public key based AND disable password based auth (for SSH) + use non-default port
Disable root account, disable root login via ssh -> only use user accounts with sudo
Setup firewall - lockdown all unused ports, keep: 22 (SSH), 80 (HTTP), 443 (HTTPS)
Setup fail2ban (ban IP's failing ssh login attempts)
Use docker for your services
Setup regular automatic updates
Setup append only backups (whole server or DB) with regular validity tests (restore the backup)
Setup notification (via E-Mail) critical events: high disk or RAM usage, unusual network traffic
Advanced: Setup disk level encryption (f.e. LUKS) - in case the hard drives will be resold: customer data can not be recovered
Advanced: Setup SELinux / AppArmor for fine-grained service permissions

Limit Disk Usage

Limit system log accumulation: journalctl --vacuum-size=200M

Limit docker dead containers / volumes / etc:

cat >/etc/systemd/system/docker-prune.service <<'EOF'
[Unit]
Description=Prune unused Docker data

[Service]
Type=oneshot
ExecStart=/usr/bin/docker system prune -af --volumes
EOF

cat >/etc/systemd/system/docker-prune.timer <<'EOF'
[Unit]
Description=Weekly Docker cleanup

[Timer]
OnCalendar=weekly
Persistent=true

[Install]
WantedBy=timers.target
EOF

Enable service:

systemctl daemon-reexec
systemctl enable --now docker-prune.timer

References

Crypto-Currencies

Sean Pedersen — Fri, 07 Nov 2025 00:00:00 GMT

Bitcoin, Ethereum and co (crypto-coins) have arrived in the mainstream - everybody and their mother has heard about it, but what does it all mean?

Snowball or the Future of Money

It is a snowball system (primary use is speculating / zero sum: every profit made equals someone else losing):

everybody holding now will eventually cash out (unless bitcoin becomes a global currency)
"Long-term holders (LTHs) now control 14.7 million BTC, 74% of the circulating supply, signaling that holding BTC has become the prevailing strategy in a maturing market." - (2)

It is a genuine new monetary system and people will not eventually cash out (signalled by how much it is used as a currency in every day life transactions).

The Reality Check

The evidence suggests it's primarily a speculative asset:

Overwhelmingly held for speculation (70-90% of Bitcoin hasn't moved in years)
Minimal retail payment adoption after 15 years
High volatility makes it poor currency
Some legitimate use: international transactions, circumventing capital controls
Stablecoins show more actual payment utility than Bitcoin

The "genuine monetary system" argument requires everyday adoption that hasn't materialized. Even crypto advocates increasingly frame it as "store of value" rather than currency - which is admitting it's speculative asset holding.

Conclusion

The technology behind Bitcoin (blockchain) is legit and might lead us to a decentralized global monetary system but it does not look like Bitcoin will be this - it is a speculative asset. Stable coins will more likely fill the role of a true digital world currency.

References

PostgreSQL Database

Sean Pedersen — Mon, 13 Oct 2025 00:00:00 GMT

Postgres is a versatile, powerful open-source relational DBMS - which means it is capable of lots of things: spanning from advanced search capabilities for text, vector search and geographic data to optimizations like partitioning and sharding. Understanding these optimization techniques can transform a moderately performing database into a high-performance system handling millions of queries per second with sub-millisecond latency.

Database Maintenance

PostgreSQL's Multi-Version Concurrency Control (MVCC) architecture creates dead tuples whenever rows are updated or deleted. An UPDATE operation actually performs a DELETE followed by INSERT, leaving the old row version behind. These dead tuples accumulate over time, consuming disk space and degrading query performance.

Standard VACUUM runs concurrently with normal operations, acquiring only a ShareUpdateExclusive lock that permits reads and writes. It marks space occupied by dead tuples as reusable within the same table but doesn't return disk space to the operating system. This process also "freezes" old tuples by marking them visible to all future transactions, preventing wraparound. Configure autovacuum with appropriate thresholds to ensure it runs frequently enough—the default 20% scale factor means a 1TB table accumulates 200GB of dead tuples before vacuuming triggers, which is far too permissive for large tables.

VACUUM FULL rewrites the entire table into a new disk file with no wasted space, returning freed disk space to the operating system. However, it requires an ACCESS EXCLUSIVE lock that blocks all operations and needs up to 2x the table's size in temporary disk space. Run VACUUM FULL only during scheduled maintenance windows for extreme bloat situations, never as routine maintenance. For a 100GB table with 90% bloat, VACUUM FULL might take hours and block all access. Instead, prevent bloat accumulation through proper autovacuum tuning.

VACUUM FULL alternative (less locking): https://github.com/reorg/pg_repack

Backups

Scaling & Optimisations

Indexes

Indexes speed up queries by generating data structures (binary trees, hash tables, etc.) that aid them - at the cost of storage and write performance. Dexter is a tool that analyses your DB's common queries and automatically suggests indexes to speed them up.

Vertical Scaling (bigger server)

TL;DR - get a beefy server

Fast IO ops, NVMe > SSD > SATA. Use RAID for even better performance
Lots of fast RAM
Beefy CPU cores (lots of em)

Check out: https://pgtune.leopard.in.ua/

PGBench

Run pgbench to eval server performance quickly.

sudo -u postgres psql -c "CREATE DATABASE testdb;"
sudo -u postgres pgbench -i -s 10 testdb  # -s 10 = 10M rows
sudo -u postgres pgbench -c 10 -j 2 -T 60 testdb # -c clients, -j threads, -T duration(sec)

TABLESPACE

TABLESPACE maps tables and indexes to specific storage volumes - useful for placing hot data on NVMe and archival data on cheaper disks (reduce costs and optimize performance).

Create new tablespace:

CREATE TABLESPACE fastssd LOCATION '/mnt/hetzner-volume';

Move existing table and index to tablespace:

ALTER TABLE table_name SET TABLESPACE fastssd;
ALTER INDEX table_name_idx SET TABLESPACE fastssd;

Table Partitioning Strategies

TODO: When to use partitions vs local indexes

Table partitioning divides large tables into smaller physical pieces while maintaining a single logical table interface. Partitioning becomes valuable when tables exceed 100GB and contain natural segmentation boundaries in query patterns. For example if the data in a table is frequently queried by a language id and the language association of items does not change frequently, it makes sense to partition the table by language id.

Range partitioning splits data by value ranges, ideal for time-series data. Create a parent table with PARTITION BY RANGE (timestamp_column), then add child partitions for each time period. Monthly partitions work well for event logs accumulating gigabytes daily. PostgreSQL 11+ supports declarative partitioning with automatic constraint management. Include the partition key in the primary key: PRIMARY KEY (event_time, id) ensures uniqueness constraints work across partitions.

CREATE TABLE events (
    id BIGSERIAL,
    event_time TIMESTAMPTZ NOT NULL,
    payload JSONB,
    PRIMARY KEY (event_time, id)
) PARTITION BY RANGE (event_time);

CREATE TABLE events_2024_01 PARTITION OF events
    FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');
    
CREATE INDEX ON events_2024_01 (event_time);

List partitioning organizes data by discrete values like geographic regions or tenant IDs in multi-tenant systems. Define explicit value lists for each partition: FOR VALUES IN ('USA', 'CANADA', 'MEXICO'). Create a DEFAULT partition to capture unlisted values, though adding new partitions when a DEFAULT exists requires scanning the DEFAULT partition to move matching rows.

Hash partitioning distributes rows evenly across partitions using a hash function, preventing hotspots when no natural ordering exists. Create with PARTITION BY HASH (customer_id) and define partitions with modulus and remainder: FOR VALUES WITH (MODULUS 4, REMAINDER 0). Hash partitioning reduced memory usage at Cubbit when they decreased from 256 to 64 partitions while maintaining query performance improvements from partition pruning.

Partition pruning eliminates scanning irrelevant partitions at query planning or execution time. Queries must include the partition key in WHERE clauses using immutable expressions. WHERE event_time >= '2024-01-01' AND event_time < '2024-02-01' enables pruning, but WHERE event_time > NOW() - INTERVAL '1 day' prevents it because NOW() is not immutable. Verify pruning with EXPLAIN—look for scans on specific partitions rather than all partitions.

Choose partition counts carefully. Too few partitions (under 10) provide minimal benefits. Too many partitions (thousands) increase query planning time exponentially and consume excessive memory. Aim for several dozen to a few hundred partitions maximum. PostgreSQL 12 improved partition handling significantly, enabling efficient management of thousands of partitions where previously 100 was the practical limit.

Index creation on partitioned tables automatically propagates to all child partitions. Create indexes on the parent table after creating all partitions for efficiency. Use CREATE INDEX CONCURRENTLY in production to avoid locking. For large partitions, create indexes concurrently on each partition individually, then attach to the parent index to maintain service availability.

Partition maintenance requires automation. Use pg_partman extension for automatic partition creation and removal based on retention policies. Detaching old partitions provides instant deletion: ALTER TABLE events DETACH PARTITION events_2020_01; DROP TABLE events_2020_01;. This avoids the bloat and performance degradation of deleting millions of rows individually. Cubbit achieved instant deletion of expired data through partition drops versus expensive DELETE operations scanning entire tables.

Performance benchmarks show partitioning trades increased write overhead for faster targeted queries. Range queries with proper partition pruning execute up to 10x faster by scanning only relevant partitions. Point queries incur 15-20% overhead due to planning complexity. Bulk loads run 20-25% slower with 400+ partitions due to partition switching costs. Optimize by loading data pre-sorted by partition key.

Horizontal Scaling (more servers)

Citus: https://github.com/citusdata/citus

Streaming replication provides the foundation for PostgreSQL horizontal scaling. The primary server continuously sends Write-Ahead Log (WAL) changes to standby servers, which replay transactions to maintain synchronized copies. Configure wal_level = replica and max_wal_senders = 10 on the primary. Create a replication user with CREATE ROLE replication_user WITH REPLICATION LOGIN. Initialize standbys using pg_basebackup to clone the primary's data directory, then configure primary_conninfo pointing to the primary server.

Asynchronous replication delivers superior performance by not waiting for standby confirmation before committing transactions. This introduces potential data loss if the primary fails before standbys receive recent transactions. Synchronous replication (synchronous_commit = on) waits for at least one standby to acknowledge WAL writes before committing, trading performance for zero data loss. Configure multiple synchronous standbys with quorum: synchronous_standby_names = 'ANY 2 (standby1, standby2, standby3)' requires any two standbys to acknowledge before committing.

Read replicas offload SELECT queries from the primary, horizontally scaling read capacity. Configure multiple standbys, then route read traffic across them using connection pooling or application-level load balancing. HAProxy provides effective round-robin distribution with health checks. Monitor replication lag with pg_stat_replication on the primary—check write_lag, flush_lag and replay_lag columns. Aim for lag under 1 second for local replicas, under 5 seconds for cross-region. High lag indicates network issues, under-provisioned replica hardware, or long-running transactions blocking WAL replay.

Logical replication enables selective table replication and cross-version replication, unlike physical streaming replication's full-cluster approach. Create publications on the source: CREATE PUBLICATION my_pub FOR TABLE users, orders. Create subscriptions on the target: CREATE SUBSCRIPTION my_sub CONNECTION 'host=source_db' PUBLICATION my_pub. Logical replication supports heterogeneous targets, enabling blue-green deployments and gradual migrations.

PgBouncer provides connection pooling, allowing 10-50x more client connections than direct PostgreSQL connections. PostgreSQL's process-per-connection model consumes 5-10MB per connection; with 1000 connections, that's 5-10GB overhead. PgBouncer runs as a lightweight proxy, maintaining a small pool of server connections while accepting thousands of client connections. Configure pool_mode = transaction for optimal concurrency—connections return to the pool after each transaction completes, maximizing reuse.

[pgbouncer]
pool_mode = transaction
default_pool_size = 25
max_client_conn = 1000
reserve_pool_size = 5
server_idle_timeout = 600

Calculate pool size using (num_cores × 2) + effective_spindle_count. An 8-core system with SSD typically needs 20 server connections. Allow 50x more client connections than server connections through pooling. Transaction mode provides 2-3x throughput improvement compared to direct connections. Limitations include incompatibility with session-level features like temporary tables persisting across transactions and SET parameters outside transactions.

Citus extension enables sharding—distributing data across multiple PostgreSQL servers. Install with CREATE EXTENSION citus, configure a coordinator node, then add worker nodes with SELECT citus_add_node('worker_host', 5432). Create distributed tables with SELECT create_distributed_table('events', 'tenant_id'), specifying the shard key. Queries filtering by shard key execute on relevant workers only, providing linear scalability.

Shard key selection critically impacts performance. Choose high-cardinality columns appearing in WHERE clauses and JOIN conditions. Multi-tenant SaaS applications shard by tenant_id, IoT systems by device_id. Avoid sequential IDs, low-cardinality columns, or frequently updated values. Colocate related tables using the same shard key to enable efficient JOINs within workers: SELECT create_distributed_table('users', 'tenant_id', colocate_with => 'events').

Schema-based sharding in Citus 12+ simplifies multi-tenant architectures. Each tenant gets a dedicated schema: CREATE SCHEMA tenant_1. Tables created in distributed schemas automatically shard by schema. This enables tenant isolation, per-tenant backups and straightforward migrations. Set search_path TO tenant_1 to route queries to specific tenants.

Search Capabilities

Full-Text Search

PostgreSQL's built-in full-text search provides text search capabilities competitive with Elasticsearch for datasets under 5 million records.

There are two complementary search modes:

Tokenized full-text search (tsvector/tsquery): language-aware matching with ranking based on term frequency and normalization. PostgreSQL provides TF‑IDF like ranking via ts_rank and cover-density via ts_rank_cd. This is not strict BM25, though it behaves similarly for many use cases. For exact BM25 scoring, consider extensions (e.g., RUM or PGroonga), otherwise ts_rank normalization options are typically sufficient.
Substring/fuzzy search (pg_trgm): fast substring, prefix and typo-tolerant matching using trigrams. This does not tokenize or stem; it matches character n-grams and is ideal for autocomplete, partial matches (%term%) and fuzzy lookups.

When to choose which:

Use tsvector/tsquery (TF‑IDF-like ranking) for language-aware search, stemming, stop words and boolean/phrase queries.
Use pg_trgm for substring/prefix matches, fuzzy lookup and autocomplete. It does not stem or understand language but excels at partial matches. Checkout biscuit index for even faster substring searches (though trading index size for speed)
Scaling: tsvector (TF‑IDF) generally scales better on large corpora (smaller, more selective indexes and lower write overhead). pg_trgm (trigram) indexes grow with text length and unique trigrams, can be much larger and heavier to maintain; prefer it for small/medium tables or short text fields (titles, usernames) and autocomplete.

Token Search (TF-IDF)

Create tsvector columns using generated columns (PostgreSQL 12+) for automatic maintenance: search_vector tsvector GENERATED ALWAYS AS (to_tsvector('english', coalesce(title, '') || ' ' || coalesce(body, ''))) STORED. This pre-computes normalized lexemes, avoiding expensive on-the-fly computation during queries. Weight different fields using setweight() to prioritize titles over body text:

search_vector tsvector GENERATED ALWAYS AS (
    setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
    setweight(to_tsvector('english', coalesce(body, '')), 'D')
) STORED

GIN on tsvector provides optimal full-text search performance, delivering query speedups of 50-100x compared to sequential scans. GIN indexes store an inverted index mapping each lexeme to matching document locations. Create with CREATE INDEX idx_search ON documents USING GIN(search_vector). For read-heavy workloads, disable fastupdate: CREATE INDEX idx_search ON documents USING GIN(search_vector) WITH (fastupdate = off). This trades slower writes for faster reads by eliminating the pending list that batches updates.

GiST indexes offer an alternative for write-heavy workloads. GiST builds tree structures with fixed-size document signatures, consuming less disk space and updating faster than GIN. However, GiST queries run approximately 3x slower and produce false positives requiring row rechecks. Choose GIN for most applications; reserve GiST for scenarios with severe write contention and limited disk space.

Query using the @@ match operator: WHERE search_vector @@ to_tsquery('english', 'postgresql & database'). Use websearch_to_tsquery() for user input—it handles quoted phrases, OR operators and minus prefixes safely: websearch_to_tsquery('english', '"full text" search -index') converts to proper tsquery syntax. Rank results with ts_rank() or ts_rank_cd() (cover density ranking considering term proximity):

SELECT title, ts_rank(search_vector, query) AS rank
FROM articles, to_tsquery('english', 'postgresql & performance') query
WHERE search_vector @@ query
ORDER BY rank DESC
LIMIT 20;

Properly optimized PostgreSQL full-text search achieves 6-10ms query times on 1.5 million records, competitive with Elasticsearch's 20ms on equivalent datasets. Keys to optimization include stored tsvector columns, GIN indexes with fastupdate disabled and appropriate weighting. Real-world implementations show fintech companies like Qonto migrating from Elasticsearch to PostgreSQL FTS to simplify their stack while maintaining similar performance.

Advanced features include phrase searches with phraseto_tsquery(), prefix matching with :* operators and proximity searches with distance operators. Highlight matching terms in results using ts_headline() with custom start/stop delimiters. Configure multiple language dictionaries per database to support multilingual content, switching configurations per query.

Extensions:

Substring / Fuzzy (trigram)

Use pg_trgm when you need partial matches (%term%), autocomplete, or typo-tolerant search. It indexes character trigrams, not tokens, and works great alongside full-text search:

-- Enable extension once per database
CREATE EXTENSION IF NOT EXISTS pg_trgm;

-- GIN trigram index: best for filtering ILIKE/LIKE/% similarity
CREATE INDEX CONCURRENTLY idx_documents_title_trgm
  ON documents USING GIN (title gin_trgm_ops);

-- Optional: GiST trigram index if you need KNN ORDER BY <-> (nearest by similarity)
-- CREATE INDEX CONCURRENTLY idx_documents_title_trgm_gist
--   ON documents USING GIST (title gist_trgm_ops);

Common queries:

-- Substring match with ranking by similarity (works well with GIN)
SELECT id, title
FROM documents
WHERE title ILIKE '%postgres%'
ORDER BY similarity(title, 'postgres') DESC
LIMIT 20;

-- Fuzzy match using trigram similarity operator
SELECT id, title
FROM documents
WHERE title % 'postgras'                 -- allows typos
ORDER BY similarity(title, 'postgras') DESC
LIMIT 20;

-- Fast KNN if using GiST trigram index
SELECT id, title FROM documents ORDER BY title <-> 'postgres' LIMIT 20;

-- Tune threshold (default ~0.3) to control fuzziness
SELECT set_limit(0.4);  -- session-level

Vector Search

There are multiple powerful extensions for vector similarity search for Postgres - check out this article for an overview with benchmarks.

Geospatial Queries (PostGIS)

PostGIS extends PostgreSQL with comprehensive geographic information system capabilities. Install with CREATE EXTENSION postgis, enabling spatial data types, measurement functions and specialized indexes. PostGIS 3.5+ requires PostgreSQL 12+ with GEOS 3.8+ and PROJ 6.1+ for full functionality. Recent versions introduced improvements like ST_CoverageClean for topology operations and enhanced SFCGAL support for 3D geometries.

PostGIS provides two core spatial types with distinct use cases. Geometry types perform planar calculations using Cartesian mathematics, appropriate for local or regional data within a few hundred kilometers. Geometry operations execute quickly but lose accuracy across large distances due to earth curvature. Geography types calculate on a spheroid representing Earth's actual shape, providing accurate great circle distances globally. Geography operations run slower but always return correct measurements in meters.

-- Geometry: fast but inaccurate for global distances
SELECT ST_Distance(
  'SRID=4326;POINT(-118.4 33.9)'::geometry,  -- LA
  'SRID=4326;POINT(2.5 49.0)'::geometry      -- Paris
);  -- Returns 122 degrees (meaningless)

-- Geography: accurate global distances  
SELECT ST_Distance(
  'SRID=4326;POINT(-118.4 33.9)'::geography,
  'SRID=4326;POINT(2.5 49.0)'::geography
);  -- Returns 9125170 meters (correct)

Spatial Reference System Identifiers (SRIDs) define coordinate systems. SRID 4326 represents WGS 84 (GPS coordinates), while 3857 provides Web Mercator used by mapping applications. Transform between projections with ST_Transform(geom, target_srid). Always set SRIDs explicitly: ST_SetSRID(ST_MakePoint(-73.9857, 40.7484), 4326). Operations require matching SRIDs or explicit transformation.

GiST indexes provide 100-1000x query speedup for spatial operations. Create with: CREATE INDEX idx_places_geom ON places USING GIST (geom). For point datasets with uniform distribution, SP-GiST indexes build 3x faster and query 1.5x faster than GiST through quad-tree partitioning. BRIN indexes suit massive, spatially sorted datasets: index sizes are 2000x smaller than GiST (24KB versus 53MB for 1M points) but require sequential access patterns.

Proximity searches use ST_DWithin() for index-aware filtering: WHERE ST_DWithin(location::geography, search_point::geography, 1000) finds points within 1000 meters. Never use WHERE ST_Distance() < 1000—this prevents index usage and scans every row. For nearest neighbor queries, use the distance operator: ORDER BY location <-> search_point LIMIT 5. This leverages GiST index K-NN functionality for efficient ordering.

-- Find restaurants within 500m
SELECT name, ST_Distance(location::geography, search::geography) AS dist
FROM restaurants
CROSS JOIN (SELECT 'SRID=4326;POINT(-73.9857 40.7484)'::geography) s(search)
WHERE ST_DWithin(location::geography, s.search, 500)
ORDER BY dist;

Spatial relationship functions enable complex queries. ST_Contains() tests if one geometry fully encloses another, useful for point-in-polygon queries: WHERE ST_Contains(neighborhood_boundary, business_location). ST_Intersects() determines overlap, ST_Within() tests containment and ST_Crosses() detects geometric crossings. Combine with aggregations for analytics: SELECT neighborhood, COUNT(*) FROM businesses JOIN neighborhoods ON ST_Contains(boundary, location) GROUP BY neighborhood.

Measurement functions calculate geometric properties. ST_Area() computes polygon areas (use geography for square meters, geometry for projected units). ST_Length() measures line lengths. ST_Buffer() creates zones around geometries: ST_Buffer(location::geography, 500) generates 500-meter radius circles around points. Subdivide complex geometries with ST_Subdivide() to improve query performance on large features.

Query optimization requires proper index usage and simplified geometries. Add bounding box pre-filters: WHERE geom && ST_MakeEnvelope(xmin, ymin, xmax, ymax, 4326) AND ST_Intersects(geom, search_area). The && operator performs fast bounding box overlap checks using the index before expensive geometric calculations. Use ST_Centroid() for approximate distance calculations on large polygons. Configure random_page_cost = 1.0 for SSD storage and effective_io_concurrency = 200 for NVMe.

Embedded (Testing)

Check out PGLite for a small embeddable WebAssembly version of Postgres - being a good alternative to SQLite and thus ideal for testing and client-side deployments of Postgres (with the limitation of accepting only one connection though).

And check out postgresql-embedded for a Rust package that embeds Postgres without limitations (supports extions).

Useful Features

EXCLUDE constraints: To avoid overlapping time slots

If you ever needed to prevent overlapping time slots for the same resource, then the EXCLUDE constraint is extremely useful. It enforces that no two rows can have overlapping ranges for the same key.

CHECK constraints: For validating data at the source

CHECK constraints allow you to specify that the value in a column must satisfy a Boolean expression. They enforce rules like "age must be between 0 and 120" or "end_date must be after start_date."

GENERATED columns: To let the database do the math

If you’re tired of calculating derived values in your app, you can let PostgreSQL handle it with GENERATED columns.

Useful Commands

Show sizes of all tables + index + toast (The Oversized Attribute Storage Technique - used for long texts / json storage):

SELECT
   relname as table_name,
   pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
   pg_size_pretty(pg_indexes_size(relid)) AS index_size,
   pg_size_pretty(pg_relation_size(relid)) AS data_size,
   pg_size_pretty(pg_table_size(relid) - pg_relation_size(relid)) AS toast_size
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;

Show sizes of all indices for a table:

SELECT
  i.indexrelname AS index_name,
  pg_size_pretty(pg_relation_size(i.indexrelid)) AS index_size,
  am.amname AS index_type,
  string_agg(a.attname, ', ' ORDER BY array_position(ix.indkey::int[], a.attnum)) AS indexed_columns
FROM
  pg_stat_all_indexes i
JOIN
  pg_class c ON i.indexrelid = c.oid
JOIN
  pg_am am ON c.relam = am.oid
JOIN
  pg_index ix ON i.indexrelid = ix.indexrelid
JOIN
  pg_attribute a ON a.attrelid = i.relid AND a.attnum = ANY(ix.indkey)
WHERE
  i.relname = 'table_name'
GROUP BY
  i.indexrelname, pg_relation_size(i.indexrelid), am.amname
ORDER BY
  pg_relation_size(i.indexrelid) DESC;

Show sizes of all indices for a partitioned table:

SELECT
  am.amname AS index_type,
  pg_size_pretty(SUM(pg_relation_size(i.indexrelid))) AS total_index_size
FROM
  pg_stat_all_indexes i
JOIN
  pg_class c ON i.indexrelid = c.oid
JOIN
  pg_am am ON c.relam = am.oid
JOIN
  pg_inherits pi ON pi.inhrelid = i.relid
WHERE
  pi.inhparent = 'table_name'::regclass
GROUP BY
  am.amname
ORDER BY
  SUM(pg_relation_size(i.indexrelid)) DESC;

Show active transactions:

SELECT pid, usename, datname, application_name, client_addr, backend_xid, xact_start, query, state
FROM pg_stat_activity
WHERE backend_xid IS NOT NULL;

Show all locks on tables:

SELECT
  COALESCE(blocking_locks.relation::regclass::text, blocking_locks.locktype) AS locked_item,
  now() - blocked_activity.query_start AS waiting_duration,
  blocked_activity.pid AS blocked_pid,
  blocked_activity.query AS blocked_query,
  blocked_locks.mode AS blocked_mode,
  blocking_activity.pid AS blocking_pid,
  blocking_activity.query AS blocking_query,
  blocking_locks.mode AS blocking_mode,
  blocking_activity.usename AS blocking_user,
  blocked_activity.usename AS blocked_user
FROM
  pg_catalog.pg_locks blocked_locks
JOIN
  pg_stat_activity blocked_activity ON blocked_locks.pid = blocked_activity.pid
JOIN
  pg_catalog.pg_locks blocking_locks ON (
    (blocking_locks.transactionid = blocked_locks.transactionid)
    OR
    (blocking_locks.relation = blocked_locks.relation AND blocking_locks.locktype = blocked_locks.locktype)
  )
  AND blocked_locks.pid != blocking_locks.pid
JOIN
  pg_stat_activity blocking_activity ON blocking_locks.pid = blocking_activity.pid
WHERE
  NOT blocked_locks.granted
ORDER BY
  waiting_duration DESC;

Kill process:

SELECT pg_terminate_backend(PID);

Export / Import

Export schema only (no indices / constraints):

pg_dump \
  -Fc \
  --schema-only \
  --section=pre-data \
  --no-owner \
  --no-privileges \
  -U postgres \
  -d your_db \
  > your_db_tables_only.dump

Export data only (rows):

pg_dump \
  -Fc \
  --data-only \
  --exclude-schema=_timescaledb_internal \
  --no-owner \
  --no-privileges \
  -U postgres \
  -d your_db \
  > your_db_data.dump

Restore (on same Posgres version):

# 1. Tables only
pg_restore \
  -U postgres \
  -d your_db \
  your_db_tables_only.dump

# 2. Data
pg_restore \
  --data-only \
  --disable-triggers \
  -j 8 \
  -U postgres \
  -d your_db \
  < your_db_data.dump

References

Postgres as Vector DB - a benchmark

Sean Pedersen — Wed, 08 Oct 2025 00:00:00 GMT

There is a flood of vector databases - which ones are actually useful? IMO extending a relational DBMS with ACID compliance and existing datasets, is for most use cases the ideal choice. Using a dedicated vector DB like (Chroma, Vespa, Turbopuffer, LanceDB, Milvus etc.) only makes sense for narrow use cases where no complicated meta-data filters are needed (e.g. just simple RAG) and data synchronisation is no issue.

So let's have a look how we can store and search vectors using Postgres - there are three notable extensions for Postgres: pgvector, pgvectorscale and vectorchord.

PGVector

Standard extension to store vectors with common medium scale ANN indices (HNSW & IvfFlat). Integration package for Python: https://github.com/pgvector/pgvector-python/

Store vectors (float 32 bit):

CREATE TABLE items (id bigserial PRIMARY KEY, embedding vector(1024));

Store half-precision (float 16 bit) vectors:

CREATE TABLE items (id bigserial PRIMARY KEY, embedding halfvec(1024));

HNSW

Hierarchical navigable small world (HNSW) is a popular graph based ANN index - delivering good retrieval and QPS performance but at the cost of longer index build times and using more RAM for it (did crash for 1 million vectors on my 16GB RAM machine).

IvfFlat

Inverted File Flat (IvfFlat) is less RAM hungry than HNSW. With caveats: IvfFlat performance degrades as vectors are deleted and added (because centroids are not updated), thus needing regular index rebuilds (if new vectors get inserted regularly).

"As data gets inserted or deleted from the index, if the index is not rebuilt, the IVFFlat index in pgvector can return incorrect approximate nearest neighbors due to clustering centroids no longer fitting the data well"

IvfFlat Binary (with reranking)

Build IvfFlat index on binarized vectors -> on query: overfetch Kx10 nearest neighbors, then rerank using float16/32 vectors to get more accurate final K-NN.

-- Mean threshold binarization (performs better than pgvector's binarizer)
CREATE OR REPLACE FUNCTION binary_quantize_mean(vec halfvec) 
RETURNS varbit AS $$
    SELECT string_agg(
        CASE WHEN val >= mean_val THEN '1' ELSE '0' END, 
        ''
    )::varbit
    FROM unnest(vec::real[]) AS val,
         (SELECT AVG(v) AS mean_val FROM unnest(vec::real[]) AS v) AS stats;
$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;

-- Step 1: Add a stored generated binary column
ALTER TABLE table_name 
ADD COLUMN vector_bin bit(512) 
GENERATED ALWAYS AS (binary_quantize_mean(vector)::bit(512)) STORED;

-- Step 2: Create index on the pre-computed binary column
CREATE INDEX IF NOT EXISTS table_name_idx_ivfflat_bin 
ON table_name 
USING ivfflat (vector_bin bit_hamming_ops) 
WITH (lists = 100);

-- Query top 100 NN using reranking
SELECT t.id
FROM (
  SELECT id
  FROM table_name
  ORDER BY vector_bin <~> binary_quantize_mean(%s::halfvec(512))::bit(512)
  LIMIT 1000
) c
JOIN table_name t USING(id)
ORDER BY t.vector <=> %s::halfvec(512)
LIMIT 100;

PGVectorScale

DiskANN

A promising ANN index that uses RAM and disk (needs fast disk - SSD / NVMe) to scale to billions of vectors, promising low RAM usage while still providing decent QPS. The problem is that the pgvectorscale index building implementation is single core right now - leading to very long index generation times.

PGVectorScale supports pre-filtering using bitfields with manual meta-data table setup (complicated / bad dev ux).

VectorChord

VChordRQ

A custom ANN index with excellent performance (combining IVF ANN index with RaBitQ quantization). Supports pre-filtering (easy to use).

VChord can be configured to not copy all vectors into the index (which is the default and pgvector also does), reducing disk usage - it can be enabled (slightly degrading performance).

VChord also supports efficient range filters (limiting results by distance instead of K nearest neighbors).

VChordG (DiskANN)

A novel addition (not prodution ready yet): custom implementation of DiskANN index combined with RaBitQ quantization.

https://blog.vectorchord.ai/vectorchord-05-new-rabitq-empowered-diskann-index-and-continuous-recall-measurement?source=more_articles_bottom_blogs

Benchmark

The benchmark should reflect realistic usage - right now it just measures index build and query times.

In the future I want to extend it:

measure insertion perormance after initial index is built
use real data embedding vectors -> simulate data distribution shift (insert many vectors after index was built)
simulate realistic complex SQL queries involving categorical and range filtering
benchmark vector scales: 100K, 1M, 10M, 100M, 1B

ANN Benchmark Results

The table below was computed using 100K CLIP Matryoshka text embeddings.

Index	Recall@100 (%)	Latency (ms)	Dim	Storage	Build (s)	Storage (MB)	Index (MB)
binary_mean_ivf_rerank	90.0	14.20	1024	float32	2.12	429.7	14.6
binary_mean_ivf_rerank	92.0	15.10	1024	float16	0.86	214.8	14.6
binary_uint8_ivf_rerank	100.0	37.67	1024	float32	7.43	429.7	98.1
binary_uint8_ivf_rerank	100.0	39.94	1024	float16	4.75	214.8	98.1
binary_uint4_ivf_rerank	100.0	17.00	1024	float32	2.77	429.7	52.5
binary_uint4_ivf_rerank	100.0	17.06	1024	float16	3.78	214.8	52.5
vchordrq	96.0	18.06	1024	float32	12.34	429.7	50.2
vchordrq	97.0	25.38	1024	float16	11.68	214.8	47.6
diskann	96.0	18.48	1024	float32	156.78	429.7	55.8

binary_mean_ivf_rerank	84.0	13.90	512	float32	0.84	214.8	8.5
binary_mean_ivf_rerank	84.0	11.70	512	float16	0.77	107.4	8.5
binary_uint8_ivf_rerank	94.0	21.77	512	float32	2.54	214.8	52.5
binary_uint8_ivf_rerank	94.0	20.03	512	float16	3.47	107.4	52.5
binary_uint4_ivf_rerank	94.0	14.34	512	float32	1.15	214.8	27.3
binary_uint4_ivf_rerank	94.0	12.03	512	float16	1.23	107.4	27.4
vchordrq	94.0	27.55	512	float32	7.70	214.8	41.5
vchordrq	94.0	23.27	512	float16	7.51	107.4	40.0
diskann	94.0	31.49	512	float32	159.26	214.8	55.8

binary_mean_ivf_rerank	73.0	11.88	256	float32	0.47	107.4	5.4
binary_mean_ivf_rerank	73.0	9.72	256	float16	0.33	53.7	5.4
binary_uint8_ivf_rerank	86.0	13.94	256	float32	1.63	107.4	27.4
binary_uint8_ivf_rerank	86.0	11.89	256	float16	1.26	53.7	27.3
binary_uint4_ivf_rerank	84.0	75.42	256	float32	1.26	107.4	14.6
binary_uint4_ivf_rerank	84.0	13.22	256	float16	1.15	53.7	14.6
vchordrq	86.0	22.14	256	float32	7.22	107.4	36.6
vchordrq	86.0	16.66	256	float16	7.16	53.7	35.9
diskann	86.0	25.21	256	float32	152.00	107.4	48.8

Optimizing Vector & Index Storage

When things scale up one should strive for efficient vector storage using:

Vector Dimensionality Reduction
- Product Quantization (PQ): reduces both dimensionality and bit representation
- Matryoshka Embeddings: superior performance vs PQ as it learns the reduction in training and not post-training
- PCA / UMAP
Scalar Quantization (Reduce bit representation per dimension)
- FP16: half precision from FP32
- binary: reduce to single bit
  - In pgvector: binary quantization will reduce any positive value to 1, and any zero or negative value to 0
  - Mean-based thresholding: binarize each dimension using its corpus-wide mean as the threshold, ensuring ~50/50 bit distribution per dimension for better information preservation and more discriminative binary codes.

We can see that using IvfFlat index on binary representation with a top-K factor of 10x (overfetching), then reranking in higher precision (float32 or float16) results in excellent recall, low vector storage costs (float16) and very fast retrieval latencies.

Though the reranking can lead to complications with queries using metadata filtering, so just using vchordrq index with 512D and float16 can also make sense.

Show me the code

Check out the code here

Conclusion

VectorChord is a good choice - providing superior performance and better developer experience (pre-filtering and better default settings). The vchordrq index is for most use cases the ideal choice as it delivers great performance and handles data distribution drift better than diskann indices. Using an ANN index only starts to make sense for big numbers of vectors (over 1 million).

References

#coding #ML

Launching DROP

Sean Pedersen — Wed, 01 Oct 2025 00:00:00 GMT

DROP is a location based social network (mobile app). It works by dropping posts (text or image) at your current location - other users can only collect a drop if they are within a radius of 50m. The idea is to promote engagement with your local community.

Now available in the App Store and soon also in the Play Store

Digital Design

Sean Pedersen — Wed, 24 Sep 2025 00:00:00 GMT

Digital design is concerned with the creation and optimization of user interfaces and experiences across digital platforms (apps, web sites, etc.). At its foundation, effective digital design balances aesthetic appeal with functional usability, ensuring that users can accomplish their goals efficiently while enjoying the interaction. Good design is invisible to the untrained eye because it just works. The design of a product sets the tone and communicates (subconsciously) values to the user.

Good Design

Feels natural — usable without thinking how. Respects the user: increases user agency; bad design extracts it.

User controls their data
- Use common file formats (for export / storage)
Information Hierarchy (pack UI elements together based on their semantics)
- Minimizes mouse movement / provide keyboard shortcuts for common inputs
Iterative complexity (progressive disclosure of options)
Minimize visual shift / movement (every moving pixel must add value)
Use rows instead of boxes for items that have numerical values to compare
Use hover highlight effects only for interactive components (links, buttons)
Emotional Design: playful / rewards
Cultural Differences: Japanese love information-dense UI

Design Trends

Trend	Real examples
AI-native UX	ChatGPT Perplexity Figma AI
Funky / unique product pages	Nothing Teenage Engineering Liquid Death Heyday Canning Co.
Bold / playful	PostHog
Neo-brutalism	Gumroad Figma
Liquid glass	Apple
Blueprint / grid aesthetic	Zed Editor Stripe
Anti-design	MSCHF Adult Swim Studio Job
Immersive motion / storytelling	Apple Nike Tesla
Editorial / mixed-media layouts	The New York Times The New Yorker BBC
Calm minimalism	Obsidian UsByNight COS Dropbox
Performance-first design	Wikipedia Craigslist McMaster

Bad Design

In order to produce good design, it is very helpful to just feel bad designs.

Dark UX:

Locking up user data (in a server / custom storage format)
Infinite scroll feeds (no ending — addictive for the user)
Fake urgency (countdown timers, "only 2 left!")
Requiring account creation before showing content
Hiding the unsubscribe button (small text / many clicks)
Forced app downloads to view mobile websites

Bad UX:

Pop-ups (visual distractions)
Deviating significantly from proven historical UI patterns
Inconsistent navigation patterns between pages
Overusage of animations (only move UI elements with a clear signal / purpose)
Loading spinners that never end / indicate no progress
Auto-playing videos with sound
Confirmation dialogs for every minor action
Non-descriptive error messages
Disabling zoom on mobile
Missing auto-focus of text input (search bars etc.)
Low contrast text (poor accessibility)

Impact

Digital interfaces are control surfaces: they don't just enable actions, they shape which actions are likely to occur. Layout, defaults, friction and feedback loops guide attention and influence our behavior — often without the user's conscious awareness.

Because of this, design is never neutral. Repeated interaction patterns become memetic: they train habits, normalize expectations and propagate values across products and cultures. An infinite feed normalizes consumption without closure; an opaque system teaches dependence; a reversible, inspectable UI teaches agency.

Bad design is therefore not just inconvenient — it is political. It encodes power asymmetries into everyday interactions by extracting our attention, data and autonomy. Good design does the opposite: it makes data control explicit and preserves optionality, to give the user the control they deserve.

References

React

Sean Pedersen — Wed, 24 Sep 2025 00:00:00 GMT

React is a very popular front-end framework allowing to modularize web UI elements in components (even allowing to create mobile apps using React Native / Expo). Components render UI elements conditioned on state variables. Components are made of other components which can be passed state using so called props.

Consider Svelte, SolidJS or MarkoJS over React for simpler and more performant (using signals) alternatives. React is more popular though and thus has a bigger eco-system and more experienced developers.

State

Only use state variables (useState) for user facing state dependent UI components. Otherwise use global vars or it will get buggy.

useEffect hook:

useEffect(() => { });          // Runs after every render
useEffect(() => { }, []);      // Runs only once after initial render
useEffect(() => { }, [count]); // Runs when count changes

Zustand is a global state manager for React which can reduce state complexity in your application by reducing property drilling (passing many state vars as props through nested components). Use it with immer for sane state management.

Use Tanstack Query to fetch data from the backend (for proper error handling, caching etc.).

https://ui.dev/why-react-query

State Management Best Practices

UI State (local): Stick with useState for ephemeral UI details (open/closed toggles, input values).

Global State (app-wide): Use context sparingly or a dedicated library like Zustand for cross-cutting concerns (auth state, theme, feature flags).

Server State (remote data): Rely on TanStack Query for fetching, caching, invalidation, and syncing with backend data. Avoid duplicating server state in global or local state.

Derive state instead of duplicating: If a state can be derived from props or other state, don’t store it separately.

References

Javascript (TypeScript)

Sean Pedersen — Thu, 18 Sep 2025 00:00:00 GMT

TypeScript is a superset of JavaScript that brings static typing, preventing many bugs found in vanilla JS code bases. TypeScript can both be used in the frontend as well in the backend (server) using NodeJS. JavaScript is a neccessary evil (that has many strange quirks), which TypeScript alleviates some of at least.

Basics

Use pnpm instead of npm for a faster and securer package manager.
Use volta instead of nvm for sane node version management.
Use vite for sane project bundling.
Try out bun for great speed and lean apps.

FrontEnd Frameworks

Use React for a popular framework, with many resources (both knowledge and developers) available. Use Svelte, SolidJS or MarkoJS for smaller and better alternatives - while SolidJS being closest to React in terms of developer experience. Read this article for an excellent comparison of frontend state management frameworks. Checkout pretext for fast text rendering. Try Three.js for 3D rendering like games and D3.js for 2D data visualization.

Promises (async)

JavaScript uses promises to handle asynchronous operations like API calls, file operations, or timers. A promise represents a value that may be available now, in the future or never and is returned by async functions (return values of async functions are always wrapped in a promise). The await keyword unwraps the value of the promise.

Promise States

A promise can be in one of three states:

Pending: Initial state, neither fulfilled nor rejected
Fulfilled: Operation completed successfully
Rejected: Operation failed

Async/Await Syntax

Modern JavaScript provides async/await syntax for cleaner asynchronous code:

async function fetchData() {
  try {
    const response = await fetch('/api/data');
    const data = await response.json();
    return data;
  } catch (error) {
    console.error('Error:', error);
  }
}

Promise Combinators

JavaScript provides several utility methods for working with multiple promises:

Promise.all(): Waits for all promises to resolve
Promise.allSettled(): Waits for all promises to settle (resolve or reject)
Promise.race(): Returns the first promise to settle
Promise.any(): Returns the first promise to resolve

// Wait for all promises
const results = await Promise.all([
  fetch('/api/users'),
  fetch('/api/posts'),
  fetch('/api/comments')
]);

References

https://www.typescriptlang.org/docs/handbook/basic-types.html
TypeScript performance tips: https://github.com/microsoft/TypeScript/wiki/Performance
https://jovidecroock.com/blog/state-vs-signals/

Erlang

Sean Pedersen — Thu, 18 Sep 2025 00:00:00 GMT

Erlang is a powerful functional programming language invented in Sweden at Ericsson in 1987, that excels at concurrency and fault-tolerance. Erlang processes are light-weight and share no memory and thus can scale easily across machines and CPU cores. Processes communicate via messages which are received in a queue that is pattern matched against the state loop (actor model). The process supervision tree allows fine-grained error management (let it crash), leading to robust production-ready apps.

Erlang is not made for math heavy computations though but can easily call C or other faster languages for that.

The BEAM (ErlangVM) powers big apps like WhatsApp and Discord, demonstrating its commercial viability.

Elixir and Gleam are notable modern dialects of Erlang that transpile straight to Erlang. Elixir offers syntactic sugar and a thriving open-source community with popular frameworks like Phoenix. Gleam is a type-safe alternative.

Show me some code

Quicksort using list comprehension:

quicksort([]) -> [];
quicksort([Pivot | T]) ->
    quicksort([X || X <- T, X < Pivot]) ++
    [Pivot] ++
    quicksort([X || X <- T, X >= Pivot]).

Fibonacci:

-module(fibo).
-author("Sean Pedersen").
-compile(export_all).

% Head recursion (inefficient: builds up exponentially growing function call stack: 2^N)
fib(1) -> 1;
fib(2) -> 1;
fib(N) when N > 1 -> fib(N-1) + fib(N-2).

% Tail recursion (efficient: behaves like a loop)
% The tail recursive optimization lets the function call stack not grow at all)
fib_(N) -> fib(N, 0, 1).
fib(1, _First, Second) -> Second;
fib(N, First, Second) when N > 1 -> fib(N-1, Second, First+Second).

Message passing:

% Ping pong across two machines/nodes in the same local network
% pingpong.erl must be compiled in working dir of the erlang shells on both machines!
% ---STARTUP INSTRUCTIONS---
% node1:
% $ erl -compile pingpong
% $ erl -sname pong -setcookie we_have_cookies
% node2:
% $ erl -compile pingpong
% $ erl -sname ping -setcookie we_have_cookies
% pong@node2> pingpong:start(ping@welle).

-module(pingpong).
-export([start/1,  ping/3, pong/0]).

ping(0, PongName, PongNode) ->
    {PongName, PongNode} ! finished,
    io:format("Ping finished~n", []);

ping(N, PongName, PongNode) ->
    {PongName, PongNode} ! {ping, self()},
    receive
        pong ->
            io:format("Ping received pong~n", [])
    end,
    ping(N - 1, PongName, PongNode).

pong() ->
    receive
        finished ->
            io:format("Pong finished~n", []);
        {ping, PingPID} -> % Note that PingPID is enough to reply (PID contains info about its node)
            io:format("Pong received ping~n", []),
            PingPID ! pong,
            pong()
    end.

start(PingNode) ->
    PongName = pong,
    register(PongName, spawn(pingpong, pong, [])),
    % Spawn function ping/3 remotely on PingNode machine
    spawn(PingNode, pingpong, ping, [3, PongName, node()]).

References

http://www.erlang.org/
https://learnyousomeerlang.com/content
https://www.erlang-in-anger.com/
Erlang project build manager: http://www.rebar3.org

Analog Computing - The future is continuous

Sean Pedersen — Fri, 12 Sep 2025 00:00:00 GMT

The artificial intelligence industry faces a growing energy crisis. Training large language models consumes massive amounts of power, while deploying AI at the edge demands efficiency that traditional digital processors struggle to deliver. This challenge has created interest in analog computing: which performs calculations using continuous physical quantities rather than discrete digital values. Recent breakthroughs suggest analog computing could deliver 10-40x energy efficiency improvements over conventional digital AI accelerators while maintaining competitive accuracy for specific applications.

Recent academic breakthroughs

The past two years have witnessed significant progress in analog AI research, with major publications demonstrating systems that work in practice rather than just theory. IBM's team published landmark results in Nature showing their 14nm analog AI chip achieving 12.4 TOPS/W chip-sustained performance, roughly 14x more energy efficient than comparable digital systems for speech recognition tasks. The chip integrates 35 million phase-change memory (PCM) devices across 34 tiles, successfully running a 45-million parameter speech recognition model with only minimal accuracy degradation.

The arXiv repository shows accelerating publication rates in analog deep learning, with comprehensive reviews examining eight different analog computing methodologies. These papers focus increasingly on practical implementations rather than theoretical possibilities, suggesting the field is maturing toward commercial viability.

Energy efficiency advantages

Analog computing's primary advantage lies in eliminating the von Neumann bottleneck - the energy-expensive separation between memory and computation that defines traditional digital architectures. In conventional AI processors, data movement between memory and compute units consumes 100-1000x more energy than the actual mathematical operations. GPUs often operate at less than 50% utilization due to memory bandwidth constraints, with external memory access dominating power consumption.

Analog systems solve this by storing neural network weights directly in memory devices that can perform computations. When IBM's PCM-based system receives input voltages, the stored conductance values multiply them automatically through Ohm's law (V×G=I), while Kirchhoff's law sums the resulting currents—effectively performing matrix-vector multiplication in a single operation. This approach enables 35 million weight parameters to execute simultaneously, achieving sub-microsecond inference times with minimal energy expenditure.

Measured results consistently show 10-40x energy efficiency improvements. IBM's analog AI chip demonstrates 12.4 TOPS/W compared to roughly 1 TOPS/W for comparable digital inference systems. Research implementations have achieved even higher efficiency, with some reporting up to 397 TOPS/W for specialized workloads, though these typically operate at reduced precision.

Digital computing retains advantages

Despite analog computing's energy benefits, digital AI accelerators maintain important advantages. Modern GPUs and TPUs achieve higher precision with controllable bit widths (FP32/FP16/INT8) and error-free computation, while analog systems typically operate at 3-4 bits effective precision due to device variations and noise. NVIDIA's H100 delivers ~2,000 TOPS peak theoretical performance and while its practical inference efficiency (~30-50 TOPS/W) lags analog systems, it handles much larger models with proven reliability.

Digital systems also demonstrate superior scalability for large models. Current analog computing demonstrations are limited to roughly 45 million parameters, while digital systems routinely handle models with 175+ billion parameters. The software ecosystem for digital AI is mature, with established frameworks, debugging tools and deployment pipelines that analog systems lack.

Manufacturing represents another advantage for digital approaches. Standard CMOS processes achieve >99% yield routinely, while analog precision requirements reduce effective yield and increase costs. Digital circuits tolerate process variations better than analog devices, which require tight control over materials and dimensions to maintain computational accuracy.

Commercial adoption

The commercial analog AI landscape shows both promising developments and notable failures. BrainChip leads commercial deployment with their Akida neuromorphic processor, claiming to be the world's first commercial neuromorphic AI producer. Their development kits are available for $799 and they recently secured licensing deals for space applications with Frontgrade Gaisler.

SynSense, a Swiss-Chinese company, has raised $10 million and serves over 100 business clients with their ultra-low-power neuromorphic processors consuming less than 500μW. Their Xylo family targets smart wearables and industrial monitoring applications where battery life is critical.

However, the sector also faces significant challenges. Rain Neuromorphics collapsed in 2024 despite $150 million in planned Series B funding and high-profile backers including Sam Altman. Mythic AI underwent major restructuring in 2023, laying off most staff before acquisition by SoftBank. These failures highlight the execution risks in translating analog computing research into viable products.

Conclusion

Analog computing for AI applications has progressed from research curiosity to demonstrated commercial capability, with measurable energy efficiency improvements of 10-40x over digital alternatives. IBM's 12.4 TOPS/W chip and BrainChip's commercial Akida processor prove the technology works in practice, not just theory.

However, significant challenges remain. Device variability limits precision to 3-4 bits, manufacturing costs exceed digital alternatives and software ecosystems require substantial development.

References

Nutrition - a strong mind needs a healthy body

Sean Pedersen — Thu, 11 Sep 2025 00:00:00 GMT

Getting healthy through food doesn't have to be complicated. Research shows that focusing on a few key areas can make a real difference in how you feel and function. Here's what the science says about eating better, based on current studies from major health institutions.

Why whole foods beat processed options

Your body works harder to digest real food, and that's actually a good thing. Studies show that whole food meals require about 50% more energy to digest compared to processed alternatives. When researchers compared people eating multi-grain bread with cheddar cheese versus white bread with processed cheese, the whole food group burned nearly twice as much energy just from digestion.

Fresh fruits and vegetables pack thousands of beneficial compounds that scientists call the "dark matter of nutrition." Garlic alone contains over 2,300 different compounds, yet most nutrition advice only focuses on a handful of them. This complexity can't be replicated in processed foods or supplements.

Large studies following over 100,000 people for 14 years found that those eating 8 or more servings of fruits and vegetables daily had 30% lower rates of heart attacks and strokes. The research consistently shows benefits for preventing diabetes, certain cancers, and maintaining healthy weight. The key is getting variety in colors and types to maximize these protective compounds.

Protein needs are higher than most people think

The standard recommendation of 0.8 grams per kilogram of body weight per day isn't enough for optimal health. This amount prevents deficiency but doesn't support muscle maintenance or growth, especially as we age.

Current research suggests most adults need 1.4 to 2.0 grams of protein per kilogram of body weight daily. For a 80 kg person, that's about 130 to 160 grams per day (depending on your muscle mass and physical activity). Older adults need even more due to something called anabolic resistance, where muscles become less responsive to protein.

The timing matters too. Getting 30 to 50 grams of high quality protein at each meal works better than cramming it all into one or two large servings. Your muscles can only use so much protein at once, so spreading it throughout the day maximizes muscle protein synthesis. Sources like eggs, meat, fish, and dairy provide complete amino acid profiles that match what your muscles need. Most people are eating "empty" food with way too much sugar, fat and carbohydrates - instead increase your healthy fat and protein intake. Try steak, fish and eggs - also good as breakfast.

Creatine benefits both muscle and brain

Creatine monohydrate stands out as one of the most researched supplements available, with over 1,000 published studies backing its benefits.

For muscle performance, creatine increases strength by an average of 8% more than placebo during resistance training. It works by boosting your muscles' energy currency (ATP) during high intensity activities. This translates to better performance in the gym and enhanced recovery between sets.

The cognitive benefits might surprise you. Meta-analyses show creatine significantly improves memory function, particularly in older adults. It also helps with processing speed and reduces mental fatigue during demanding tasks. Single doses can improve cognitive performance during sleep deprivation, with effects lasting up to 9 hours.

The standard protocol involves 3 to 5 grams daily for maintenance (I recommend up to 7 grams daily), though you can load with 20 grams daily for the first week to see faster results. Consistency is key for good results - so establish a daily morning routine. Safety data spanning over 20 years shows no adverse effects at recommended doses. The only consistent side effect is temporary water weight gain in the first week as muscles store more water. If you know you will consume alcohol at a given day, be extra careful to consume enough water, as alcohol dehydrates you.

Creapure is a known brand that produces high-quality creatine (regular quality testing -> no heavy metals or other unwanted additions).

Magnesium fills a common gap

Nearly half of Americans don't consume enough magnesium, yet this mineral participates in over 300 enzyme reactions in your body. The shortfall creates opportunities for meaningful improvement through supplementation.

Magnesium plays essential roles in muscle function, sleep quality, blood pressure regulation, and energy production. Research shows supplementation can modestly reduce blood pressure, help prevent migraines, and improve sleep duration and quality. The mineral acts as a natural calcium antagonist, helping muscles relax after contraction.

Not all magnesium supplements work equally well. Forms bound to organic compounds like citrate or glycinate absorb much better than cheaper options like magnesium oxide. Studies show oxide has only about 4% absorption rates, while citrate and glycinate forms absorb significantly better and cause less digestive upset.

Starting with 200 to 400 milligrams daily works well for most people. Take it with food if you experience stomach irritation, and be aware that doses over 350 milligrams from supplements can cause loose stools in some individuals.

Vitamin D3 addresses widespread deficiency

Vitamin D deficiency affects an estimated 50% of the global population, with rates even higher in northern climates and among people who spend most of their time indoors. Unlike other vitamins, your body produces vitamin D only when skin is exposed to UVB radiation from sunlight, making it technically a hormone rather than a traditional vitamin.

The health implications of adequate vitamin D status extend far beyond bone health. Research shows vitamin D receptors exist in nearly every tissue in your body, influencing immune function, muscle strength, cardiovascular health, and mood regulation. Large observational studies link higher vitamin D blood levels with reduced risks of respiratory infections, autoimmune diseases, and certain cancers.

Meta-analyses demonstrate that vitamin D supplementation can reduce the risk of acute respiratory infections by 12% overall, with even greater benefits for those who were deficient to begin with. The vitamin plays crucial roles in immune system regulation, helping your body mount appropriate responses to pathogens while preventing excessive inflammation.

For muscle function and bone health, vitamin D helps your body absorb calcium efficiently and supports muscle protein synthesis. Studies show that correcting vitamin D deficiency can improve muscle strength and reduce fall risk in older adults by up to 19%.

Most adults need 1,000 to 4,000 IU (25 to 100 micrograms) of vitamin D3 daily to maintain optimal blood levels between 30 to 50 ng/mL (75 to 125 nmol/L). The exact amount depends on factors like body weight, skin color, geographic location, and current vitamin D status. Vitamin D3 (cholecalciferol) is preferred over D2 (ergocalciferol) as it raises blood levels more effectively.

Taking vitamin D with a meal containing fat improves absorption since it's a fat-soluble vitamin. Safety margins are wide - toxicity only occurs at extremely high doses (typically above 10,000 IU daily for months), and regular blood testing can help optimize your individual dosing.

What to avoid

Two major sources of health issues hide in plain sight: chemicals from plastic food containers and regular consumption of sodas or sweetened drinks.

Chemicals like BPA and phthalates leach from plastic containers into food and drinks. These compounds act as endocrine disruptors, mimicking hormones in your body. Consumer Reports found phthalates in 84 out of 85 food products they tested, with levels in some items exceeding those typically found in fast food.

The science links these chemicals to cardiovascular problems, cognitive decline (diabetes risk + anxiety) and reproductive health issues. While regulatory agencies maintain current safety levels, the research continues to raise concerns about cumulative exposure over time.

Sugar sweetened beverages create multiple health problems. Beyond the obvious weight gain from liquid calories that don't satisfy hunger like solid food, regular soda consumption increases diabetes risk through high glycemic loads and fructose metabolism effects. Studies following hundreds of thousands of people show clear dose response relationships, where more soda equals higher disease risk.

The dental damage alone should give pause. Sodas have a pH of 2 to 3, creating an acidic environment that erodes tooth enamel while feeding harmful bacteria. Research shows even moderate consumption increases cavity risk by 57%. Simply put: drink water and tea, avoid the rest.

Putting it together

These five areas represent changes backed by solid research that can meaningfully impact health. Focus on eating more whole foods and adequate protein, limit exposure to plastic chemicals and sugary drinks, and consider creatine and magnesium supplementation if appropriate for your situation.

The beauty of these approaches lies in their simplicity and strong evidence base. You don't need perfect implementation to see benefits. Start with one or two areas that resonate most, make consistent changes, and build from there. Your body will respond to these evidence based improvements in nutrition over time.

References

YouTube: Alarming Effects From Microplastics on Human Health - Anton Petrov

Understanding Game Theory Fundamentals

Sean Pedersen — Wed, 10 Sep 2025 00:00:00 GMT

We are all players in games, though mostly unaware. Knowing the rules (what is possible) and reading other players' intentions are the two skills that separate good strategists from reactive ones. Game theory gives those instincts a mathematical backbone.

More precisely, game theory is a framework for analyzing decisions where outcomes depend not just on your choices, but on what others choose at the same time. This makes it useful for anything from corporate pricing wars to international treaties to everyday negotiations.

The core insight is this: good strategy requires anticipating responses, not merely choosing what looks best in isolation. Once you see that, you start finding strategic structure everywhere.

Core concepts

Game theory grew from the work of mathematician John von Neumann and economist Oskar Morgenstern, whose 1944 book Theory of Games and Economic Behavior laid the foundations. John Nash later extended the field with the equilibrium concept that bears his name.

A "game" in this context means any situation where multiple players make choices that affect everyone's outcomes. Players can be individuals, companies, countries, or biological organisms.

The most important concept is the Nash equilibrium. It describes a stable state where no player can improve their outcome by changing strategy alone. Everyone is playing their best response to what everyone else is doing.

Consider a hypothetical duopoly: two airlines serving the same route. If both charge high fares, both profit. If one cuts prices, it steals passengers. If both cut, neither gains much. The stable outcome (where neither airline benefits from moving first) can be interpreted as a Nash equilibrium. Real markets show patterns consistent with this structure, but calling any specific market a Nash equilibrium requires careful analysis of actual costs, margins, and competitive options.

Dominant strategies are choices that are better regardless of what opponents do. Advertising is a textbook example. Coca-Cola benefits from advertising whether Pepsi advertises or not. In practice, most strategic choices are only dominant under certain conditions, not universally.

Payoff matrices map these interactions visually. Each cell shows the outcome for every combination of choices, making the structure of incentives visible at a glance.

Types of strategic situations

Cooperative vs. non-cooperative games divide situations where binding agreements are possible from those where they are not. International climate negotiations aim to create cooperative frameworks. Without enforcement, individual countries may still defect for short-term gain, pushing the game toward non-cooperative dynamics.

Zero-sum games are those where one player's gain equals another's loss. Poker is a clean example. Most real-world situations are not zero-sum: trade, for instance, can benefit both sides.

Sequential vs. simultaneous games differ in timing. In sequential games, you observe the other player's move before responding. In simultaneous games, you act without knowing what others chose. A sealed-bid auction is roughly simultaneous. A patent race is sequential.

The prisoner's dilemma and its extensions

The prisoner's dilemma is the most studied game in social science. Two players each choose to cooperate or defect. Defecting is individually rational (it is a dominant strategy), but if both defect, both end up worse than if both had cooperated.

A clear business example: two competing firms advertising the same product. Each benefits from advertising whether the other does or not. So both advertise heavily, spending more than they would under mutual restraint. The outcome is a Nash equilibrium, but it is worse for both than cooperation would be. This pattern appears in ad spending between Coca-Cola and Pepsi, between competing airlines, and in many other industries.

Legal plea bargaining follows a similar structure. When multiple defendants face the same charges, each may have an individual incentive to cooperate with prosecutors regardless of what the others do. This can push everyone toward deals even when coordinated silence might have served the group better. The logic is structural: it applies across many multi-defendant cases, not to any single one.

The tragedy of the commons extends this to shared resources. Each person has an incentive to exploit a common resource (a fishery, a highway, clean air) beyond what is collectively optimal. The result is depletion or congestion. Property rights, quotas, or taxes often realign individual incentives with collective outcomes.

Evolutionary stable strategies come from biology. A strategy is evolutionarily stable if a population using it cannot be invaded by a mutant playing something different. This explains why animal conflicts tend toward displays rather than fights: pure aggression destabilizes a population over time.

What game theory cannot do

Game theory does not predict human behavior perfectly. People cooperate even when defection is dominant (as public goods experiments show), signal to build reputations, and care about fairness. The models are simplifications. A Nash equilibrium describes a stable outcome, not necessarily the one that will occur.

But the framework is still useful. It forces you to ask: who are the players, what can they do, and what do they want? Once the incentive structure is visible, decisions that looked simple often reveal hidden tensions.

We are all playing games. Most of us just never learned to see the board.

References

Python

Sean Pedersen — Tue, 19 Aug 2025 00:00:00 GMT

Just some notes on how to use Python effectively. Python strives to be simple and concise and is a good choice for rapid software development. The average execution speed is rather slow, which is ok because it nicely interacts with C, C++ and Rust which are fast. Another major selling point is the rich ecosystem of useful packages, spanning web development, data science and much more.

General Tips

Keep functions pure (no side effects) if possible
Use early returns in functions to avoid deep nesting
Use type hints and a type checker (ty) with pre-commit hooks
Use ruff - a fast linter / formatter
Use pathlib module for dealing with file system paths
No magic numbers (use expressive variable names e.g. waiting_time_ms)
Use f-strings for formatting strings
Validate variable types from external (untrustworthy) inputs, e.g. user input, web requests
- try attrs and cattrs instead of pydantic
Use caching for heavy computations
Use pytest for unit testing

Package Management

Use uv (preferred and popular) or pixi (can install conda packages - useful for GPU/CUDA stuff) for fast and sane package management.

Interactive Development

Great for prototyping, one-off analysis scripts and literate programming.

Jupyter Notebook and Marimo.

Web Development

Use FastAPI to create clean and simple REST API's supporting both synchronous and asynchronous routes
Use niquests (the successor of requests) for network requests (being fast, supporting async, http/2 & much more)

CLI

For creating CLI: cyclopts
For formatting console output: rich
Progress bar: tqdm

Desktop GUI Apps

PyTauri / PyLoid / PyWebView

Concurrency

Python is a single threaded language with a Global Interpreter Lock (GIL). Meaning only multi-processing enables real parallel execution of non IO code. Multi-threading or async in Python thus only allows for concurrent IO operations (network / file system read and writes).

Python 3.13 has added experimental support for a no-GIL build flag, enabling true multi-threading support, which may become the default in the future.

On multi-threading: https://glyph.twistedmatrix.com/2014/02/unyielding.html
Good ref on Python multiprocessing: https://pythonspeed.com/articles/python-multiprocessing/

Multi-Processing

Use joblib for sane multi-processing. Note that multi-processing should only be used to parallelize very CPU heavy tasks, since the overhead of starting processes is very high (always benchmark).

from math import sqrt
from joblib import Parallel, delayed

# Runs in 4 processes in parallel, preserves input order
results = Parallel(n_jobs=4)(delayed(sqrt)(i ** 2) for i in range(16))
print(results)

Async

I do not like async syntax wise (just personal preference) - I prefer multi-threading to speed up heavy IO tasks (relevant HN thread).

import asyncio
import niquests

async def fetch_multiple_urls(urls):
    async with niquests.AsyncSession() as client:
        tasks = [client.get(url) for url in urls]
        responses = await asyncio.gather(*tasks)
        return responses

Generators

For efficient (lazy / easy on RAM) code

# Loads entire file into RAM
def read_large_file_bad(filename):
    with open(filename) as f:
        return [int(line.strip()) for line in f]

# Only keeps one line in memory
def read_large_file_good(filename):
    with open(filename) as f:
        for line in f:
            yield int(line.strip())

# Memory efficient processing (file can be bigger than RAM)
total_sum = 0
for number in read_large_file_good("huge_file.txt"):
    total_sum += number

SQLite

SQLite3 support is built into the Python standard lib and a simple option to embed, store and analyze relational data.

When creating tables always use the STRICT keyword, to enfore type consistency on INSERT and UPDATE operations. This prevents ugly typing bugs that are possible - as Python does not guarantee type consistency at runtime.

https://bigcodenerd.org/blog/sqlite-type-checking-gochas/

Postgres

Postgres is very versatile and powerful DBMS. Use it with psycopg and a docker image.

from contextlib import asynccontextmanager
from fastapi import Depends, FastAPI
import psycopg_pool
import psycopg

conn_string = "postgres://postgres@localhost"

pool = psycopg_pool.AsyncConnectionPool(conn_string, open=False)

@asynccontextmanager
async def lifespan(app: FastAPI):
    await pool.open()
    yield
    await pool.close()

app = FastAPI(lifespan=lifespan)

async def get_conn():
    async with pool.connection() as conn:
	    yield conn

@app.get("/visit/")
async def add_visit(conn = Depends(get_conn)):
    async with conn.cursor() as cursor:
        # Run our queries
        await cursor.execute("insert into visits(timestamp) values (now())")

    return {"message": "Visit logged"}

Docker

Bundle your apps and make them reproducible using docker (with uv or pixi).

Logging

Use loguru (comes with a multi-processing queue that just works)

Performance

Use a profiler (pyinstrument) to find slow or RAM consuming code paths.

Use C / C++ / Rust / Zig / Mojo etc. for performance critical code or try PyPy and Cython first. Or check out a transpiler https://github.com/py2many/py2many / https://github.com/paiml/depyler.

Search is hard

Sean Pedersen — Sat, 16 Aug 2025 00:00:00 GMT

Search is the process of finding useful matches in a (large) set of objects (search corpus). A search engine is only as good as the representations it uses to match queries to the objects. To find things fast, one orders the search corpus by structuring the objects in a way (a graph) that makes it easy to navigate - this is called an index. A useful search engine understands the users intent and mirrors how humans associate information (semantics, syntax, sound, visual), so the results feel intuitive.

Use Cases

Search Interfaces (for books, web pages, images, etc.)
Recommender Systems (for similar movies, products, etc.)
Retrieval Augemented Generation (provide relevant context for LLM's)

Representations

The key of every search system is to build useful representations of your search corpus, so you can (quickly) find what you search for. Thus one should spend spend good time on finding the right representation for the search problem and how to enrich it with useful information.

Granularity (chunking)

Depending on the use case or query, different levels of information granularity of the search corpus are useful. Searching for a whole book / chapter / paragraph / sentence / word / letter or video / scene / image / objects / textures / colors.

Chunking projects:

Tricks:

Use LLM to enrich your search corpus with relevant metadata (do the work before the search)
- Ask LLM to provide a descriptive title and a short summary for each text document
- Prepend title and summary / section to every text chunk before embedding

Token Based Representations

Word Level (BM25 / normalized TF-IDF)
- Tokenization: standard words
- Purpose: relevance scoring for full-text search
- Strengths: scales efficiently to very large text corpora
- Limitation: cannot match partial words, abbreviations, or morphological variants
Subword Level (Corpus-Derived Subword Index – CDSI)
- Tokenization: greedy longest-match subwords mined from frequent corpus substrings
- Purpose: partial-word and compound-word matching in full text
- Strengths: captures abbreviations and morphemes (e.g., "neural net" matches "neural network"), smaller posting lists than tri-grams, deterministic and interpretable
- Limitation: requires offline vocabulary build and periodic rebuilds
Character Level (Tri-gram Index)
- Tokenization: overlapping 3-character sequences
- Purpose: substring and fuzzy search
- Strengths: handles typos, partial matches, and short strings (filenames, codes, IDs)
- Limitation: high space usage; mainly useful for short strings rather than full text

Method	Space	Substring coverage	Fuzzy/typo tolerance	Best use case
Tri-grams	High	All	Limited	Short strings, small-medium corpora
Variable n-grams	Medium	Most	Limited	Medium-length strings
Prefix/Suffix	Low-Med	Start/End only	No	Autocomplete, filenames
Permuterm	Medium-High	Arbitrary	No	Wildcard/substring search
Q-gram + Hashing	Low	Most	Approximate	Large-scale, fuzzy search
Suffix Array / Tree	Medium	All	Exact	Exact substring matching, static datasets

Neural Representations

Create representations using deep neural networks - typically dense vectors (embeddings).

Multi-Modal (shared) Embedding Spaces (CLIP)
Dense vs Sparse embeddings (SDR)
Query vs document prefix for embedding models optimized for search
Visual Document Embeddings
- Instead of embedding sequences of tokens (often words), embed images of pages directly -> this helps to process complex documents containing tables, graphs and imagery
- https://huggingface.co/collections/nomic-ai/nomic-embed-multimodal-67e5ddc1a890a19ff0d58073
- https://huggingface.co/jinaai/jina-embeddings-v4
Text Embeddings (chunking strategies)
- relevant segment extraction (RSE) - https://d-star.ai/dynamic-retrieval-granularity: just use paragraph-sized chunks (~200 tokens), prepend global document context (title, section) -> embed and store position in document (line number / page position). embed query -> find nearest chunks -> compute continuous chunk segments in documents
- late-interaction: do not compress tokens into single vector, instead vectorize every token (using contextual embedding model like ColBERT, so token vectors contain contextual information) and match using the MaxSim operator
  - Pros: much more granular results (excel at out-of-domain, long-context), good interpretability because of token-level matching
  - Cons: heavy computation -> search takes longer, storing cost -> n token vectors per document vs 1 document vector
- embed a document by computing sentence level dense embeddings. Transform these sentence embeddings into SDR that are stacked to preserve information. Now match a query with the same technique to find relevant docs.
- use LLM to enrich representations
  - summarize document and embed it (harnesses LLM to solve long-context problem)
  - infer questions from the chunk that it answers and embed them

Searching

Once our set of objects is transformed into useful representations (vectors), we can finally search. We need to transform the user query into the same representation to compare it with the existing set of vectors. The most common metric for this is the cosine distance (angle between two vectors -> 0: identical / very similar vectors, 1: orhogonal / very different vectors).

Semantic Search Result Scoring

For most semantic search and RAG systems:

Retrieve a candidate set (typically Top 20–100 results) using vector similarity.
Apply score gap (elbow) analysis to identify the natural relevance cutoff by finding the largest meaningful drop between adjacent scores.
Enforce sensible bounds (for example, return at least 3 and at most 10 results) to avoid edge cases where too few or too many matches are returned.
Optionally rerank the remaining candidates using a cross-encoder or LLM reranker and apply a final relevance threshold.

This approach combines the efficiency of vector search with a dynamic, query-specific relevance threshold. Unlike fixed Top-K retrieval or static similarity thresholds, it adapts to the score distribution of each query, returning all strongly related matches while filtering results that fall outside the primary relevance cluster.

Reranker Models

Serve the purpose to run a complex (big) model (specialised or LLM) to filter the results (candidates) of the vector similarity search further by computing another similarity score using the reranker model with the query for each candidate.

Modern Search Pipelines

Depending on your requirements (accuracy and time to response) pipelines of different complexity are possible.

Query Expansion

classical: use first k hits of og query to find more relevant matches
BM25: Augment the BM25 query with synonyms or reformulations from a predefined dict.
LLM based: create hypothetical document embeddings (HyDE) by generating a possible query result
- TODO: generate k possible HyDEs for ambiguous queries, prompt to just generate document keywords to reduce LLM output latency

Simple
fast but may contain irrelevant results

Document -> chunk -> embed (index)
Query -> embed -> search on chunks -> results

Reranker

Document -> chunk -> embed (index)
Query -> embed -> search on chunks -> rerank (filter) -> results

LLM (Agentic Search)

Document -> LLM (generate descriptive title + summary) -> chunk -> embed (index)
Query -> expand / rewrite query using LLM -> embed -> search on chunks -> rerank (filter) -> results

Tree-Organized Retrieval

An alternative paradigm that builds a hierarchical tree from a document and lets an LLM navigate it via tree search. Retrieval becomes reasoning over structure rather than similarity over vectors. Also marketed as "vectorless RAG".

RAPTOR (ICLR 2024): recursively cluster and summarize chunks bottom-up to build a multi-level summary tree; retrieve across abstraction levels
PageIndex: generate a "table of contents" tree, then perform agentic tree search with an LLM to locate relevant sections
- No vector DB, no chunking - sections are natural document divisions
- Traceable retrieval (page and section references) instead of opaque nearest-neighbor matches
- Trades retrieval latency and LLM cost for relevance on long, structured professional documents (where similarity ≠ relevance)
- Reported sota 98.7% accuracy on FinanceBench

Scaling Things Up

All of the following techniques trade retrieval accuracy for speed / storage costs. For production use cases, using a Vector DB is the right choice.

Approximate Nearest Neighbor (Search Index) - benchmark
- HNSW: builds a hierachical graph, good data drift handling -> high RAM usage
- IVFFlat: low RAM usage, bad data drift handling -> needs frequent rebuilds
- DiskANN: low RAM usage, achieved by using disk (needs fast disk read)
- TODO (eval): https://github.com/yichuan-w/LEANN
Vector Dimensionality Reduction
- Matryoshka Embeddings: dim. red. baked into training via loss function
- PCA (linear)
- t-SNE / UMAP (non-linear)
Vector Quantization
- Reduce bit representation (f.e. to INT8 instead of FP32)
- Near-lossless compression for unit-norm embedding vectors using spherical coordinates - code

Excellent article detailing an efficient vector search pipeline: https://huggingface.co/blog/embedding-quantization

Results Diversification

It often makes sense to present the user for general queries not only the top nearest neighbors but a diverse set of results, so the ambiguity of the query is reflected. Ideally the user can then narrow down (load more relevant results) based on the diverse result set.

Evaluation Metrics

As you see there are many knobs to tune a modern search pipeline and thus we need hard evaluation metrics to judge the quality of our search pipeline. Creating a custom dataset that mirrors our real world search use case as closely and diversely as possible is essential to improving our search. Relevant post - recommends optimizing for recall with K > 100.

Precision@K

What: Fraction of top K results that are relevant
Formula: Relevant results in top K ÷ K
Use when: You only care about the quality of what users see
Limitation: Doesn't measure how many relevant documents were missed

Recall@K

What: Fraction of all relevant documents found in top K
Formula: Relevant found in top K ÷ total relevant documents
Use when: Missing relevant information is costly
Limitation: Requires knowing all relevant documents

F1 Score@K

What: Balance between precision and recall
Formula: 2 × (Precision × Recall) ÷ (Precision + Recall)
Use when: Comparing systems with different precision/recall trade-offs
Key insight: Penalizes systems that optimize only one metric

Ideas to Explore

User steered semantic search by selecting N matches and finding the common subspace in their embeddings

References

#coding #ML

Biological Computing

Sean Pedersen — Mon, 11 Aug 2025 00:00:00 GMT

Biological computing using living neurons has crossed from science fiction into commercial reality. The first biological computers are now available for purchase, using lab-grown human brain cells to process information with high energy efficiency. These systems represent a fundamental shift from traditional silicon-based computing toward hybrid biological-digital architectures that leverage billions of years of neural evolution.

This emerging field, termed Organoid Intelligence, uses the computational power of actual brain tissue grown from human stem cells. Unlike neuromorphic chips that simply mimic neural behavior, biological computers employ real neurons that form functional networks, learn through experience and process information through natural synaptic connections.

Basic Functionality

Biological computing systems work by growing brain organoids from human induced pluripotent stem cells in laboratory conditions. These three-dimensional neural cultures, typically 0.5 to 5 millimeters in diameter, contain between 10,000 and 800,000 living neurons that spontaneously develop electrical activity and complex oscillatory behavior similar to human brain tissue.

The technical architecture integrates these living networks with multi-electrode arrays containing 8 to 384 electrodes that both stimulate and record neural activity. Advanced microfluidic life support systems maintain organoid viability for up to 100 days, a significant improvement from initial lifespans measured in hours. Communication occurs through electrical stimulation and chemical signals using neurotransmitters like dopamine and serotonin.

The systems leverage natural neural plasticity mechanisms for learning. Correct responses receive predictable electrical patterns while errors trigger chaotic stimulation, mimicking reward-based learning in biological brains. Information storage occurs through structural and functional changes in neural networks, enabling pattern recognition and memory formation through synaptic plasticity.

Early Adoption

Biological computing has moved beyond laboratory demonstrations into practical applications across healthcare, research, and technology sectors, though adoption remains in early stages.

Pharmaceutical companies are exploring biological computing for drug discovery and neurological disorder research. The technology enables disease modeling for conditions like Alzheimer's, epilepsy, and Parkinson's disease using human-relevant neural responses. This approach offers alternatives to animal testing while providing more accurate data for pharmaceutical development.

Healthcare applications include medical device development, brain-computer interface research, and personalized medicine approaches. The systems enable real-time monitoring of drug effects on neural networks and enhanced understanding of brain disease mechanisms.

Current limitations present significant adoption barriers. Biological components survive only 100 to 180 days on average, requiring regular replacement. Systems remain limited to thousands or hundreds of thousands of neurons, far below brain complexity levels. High costs, with CL1 units priced at $35,000, limit accessibility for many potential users.

Regulatory and ethical concerns constrain adoption. Ongoing debates about potential consciousness in biological computing systems require new ethical frameworks governing the use of human neural tissue. Using biological neural networks opens up many hairy ethical concerns and has an eerie similarity to the plot of The Matrix. Before we understand the emergence of (self-)cosciousness, we should not rush the adoption of this bio-technology as it may be an inherently evil application.

Applications for Artificial Intelligence

Biological computing systems may offer critical advantages over classical deep learning approaches, addressing fundamental limitations. These living neural networks demonstrate superior generalization abilities, natural robustness against adversarial attacks and adaptive learning mechanisms.

Generalization represents the most significant advantage of biological neurons. Classical deep learning models often fail catastrophically when encountering data distributions different from their training sets. A neural network trained to recognize cats in photographs might completely fail when presented with cartoon drawings of cats. Biological neurons, however, demonstrate remarkable ability to extract meaningful patterns and apply learned concepts to novel situations through natural abstraction mechanisms developed over millions of years of evolution.

Research at Johns Hopkins University showed that brain organoids could adapt their learned behaviors to new environmental conditions within hours, while equivalent artificial neural networks required complete retraining. The biological systems maintained performance across different stimulus patterns by leveraging inherent plasticity mechanisms that classical AI lacks.
Adversarial robustness emerges naturally in biological systems. Traditional deep learning models are notoriously vulnerable to adversarial examples, where tiny imperceptible changes to input data cause complete misclassification. These attacks exploit the brittle decision boundaries learned by artificial networks. Biological neurons, by contrast, process information through noisy, analog mechanisms that naturally resist such manipulations.

FinalSpark demonstrated this robustness by showing that organoid-based pattern recognition systems maintained accurate classification even when input signals were corrupted with noise levels that completely broke equivalent digital neural networks. The biological systems' inherent stochasticity and redundant processing pathways provide natural defense against adversarial perturbations.

Continual learning without catastrophic forgetting represents another key advantage. Classical AI systems suffer from catastrophic forgetting, where learning new tasks erases previously acquired knowledge. This limitation requires complex architectural modifications and careful training procedures to overcome. Biological neurons naturally support lifelong learning through synaptic homeostasis and distributed memory representations.

Cortical Labs' experiments showed that organoids could sequentially learn multiple pattern recognition tasks while retaining performance on earlier learned behaviors. The biological systems demonstrated graceful interference patterns rather than the complete knowledge destruction seen in traditional neural networks.

Key Startups

Cortical Labs, founded in 2019 as a spinout from Monash University, leads commercial biological computing development. The Australian company pioneered the DishBrain technology that learned Pong. CEO Dr. Hon Weng Chong and Chief Scientific Officer Brett Kagan have positioned the company at the forefront of wetware-as-a-service offerings. Cortical Labs launched the CL1 in March 2025 - the world's first commercial biological computer. Priced at $35,000 per unit, the system contains 800,000 lab-grown human neurons with integrated life support maintaining cell viability for six months.

FinalSpark, established in 2014 by Dr. Fred Jordan and Dr. Martin Kutter, developed the first remote-access biological computing platform. The Swiss company, currently seeking $50 million in Series A funding, has created a unique dopamine-based reward system for training organoids and extended operational lifespans from hours to over 100 days.

Koniku, founded in 2015 by Nigerian neuroscientist Dr. Oshiorenoya Agabi, focuses on biological detection systems. With $1.8 million in funding and partnerships with Airbus, the company developed Konikore processors that combine biological neurons with silicon for detecting explosives, drugs, and diseases. Their systems achieve parts-per-billion sensitivity with response times under 10 seconds.

Conclusion

Biological computing using living neurons has transitioned from theoretical research to commercial reality, offering a sustainable path toward more intelligent, adaptive and energy-efficient computing systems. While technical challenges around lifespan, scalability, and standardization remain significant, the field demonstrates extraordinary potential for revolutionizing drug discovery, neuroscience research, and artificial intelligence applications.

The convergence of stem cell technology, advanced electrode interfaces and artificial intelligence has created a new computational paradigm that leverages natural neural evolution. As these systems mature and scale, they promise to address the energy crisis in computing while unlocking new forms of machine intelligence that complement traditional silicon-based approaches. The first commercial biological computers mark the beginning of a transformation that could fundamentally reshape how we approach computation in the coming decades, though the ethics of it all remains unclear at best and evil at worst.

References

Generalization and Biological vs Artificial Neural Networks

Adversarial Robustness in Biological vs Artificial Systems

Catastrophic Forgetting and Continual Learning

Energy Efficiency and Biological Computing

Photonic Computing

Sean Pedersen — Thu, 07 Aug 2025 00:00:00 GMT

Photonic computing uses light (photons) instead of electrons to perform calculations, offering faster processing speeds and lower energy consumption than traditional electronic processors. Since photons travel at the speed of light and generate minimal heat, photonic systems can perform massive parallel operations with reduced thermal management requirements - making photonic computing a very promising development.

How Photonic Computing Works

Photonic computers manipulate light signals using various optical components:

Optical Logic Gates: Instead of transistors switching electrical signals, photonic systems use devices like Mach-Zehnder interferometers, ring resonators or nonlinear optical materials to perform logical operations on light beams. These can implement AND, OR, NOT and other fundamental operations of computing.

Wavelength Division Multiplexing: Multiple data streams can be encoded on different wavelengths of light traveling through the same optical waveguide simultaneously, enabling massive parallelism that's difficult to achieve with electrons.

Photonic Integrated Circuits: Similar to electronic chips, these integrate lasers, modulators, waveguides and photodetectors on single substrates, typically made from silicon, indium phosphid or lithium niobate.

From Lab to Market

Photonic computing transitioned from laboratory experiments to commercial systems in 2025. Multiple systems now deliver practical AI acceleration with performance that matches or exceeds traditional electronic alternatives while consuming 30 to 1,000 times less energy.

Key Breakthroughs

MIT's Integrated Processor: In December 2024, MIT researchers demonstrated the first fully integrated photonic processor capable of performing all key deep neural network computations optically on a single chip. The system achieves classification tasks in less than 0.5 nanoseconds with over 92% accuracy.

Lightmatter's Achievement: Demonstrated the first photonic processor running state-of-the-art neural networks including ResNet, BERT, and deep reinforcement learning algorithms without modifications. Their six-chip package delivers 65.5 trillion operations per second while consuming only 78 watts of electrical power plus 1.6 watts of optical power.

Tsinghua's Taichi Chip: Achieved 160 TOPS/W energy efficiency with 91.89% accuracy on image recognition tasks, representing over 1,000 times more energy efficiency than electronic counterparts.

Startup Success

Several startups have achieved significant milestones:

Lightmatter: Achieved $4.4 billion valuation after raising $400 million in 2024
PsiQuantum: Raised $940 million from the Australian government in 2024
Q.ANT: Secured €62 million Series A and became the first company to ship commercial photonic processors as standard PCIe cards
Photonic Inc.: Received $100 million investment from Microsoft

AI Applications

Photonic computing delivers three key advantages for AI workloads:

Massive Parallelism: Matrix multiplication operations can be performed passively
Ultra-low Energy Consumption: Up to 1,000x more power efficienct than digital chips
Speed: 2 to 3 orders of magnitude faster than digital chips

Real-world applications include computer vision for autonomous vehicles, natural language processing with Transformers, wireless communications for 6G systems and scientific computing for climate modeling.

Conclusion

Photonic computing achieved commercial viability in 2025, offering compelling alternatives to traditional electronic processors for specific AI acceleration tasks. While it won't replace electronic systems entirely, it has become essential for heterogeneous computing architectures where energy efficiency, processing speed, and parallel operations provide real value.

The technology has moved beyond promising research to deliver practical solutions for real-world AI workloads, marking the beginning of a new era in computational efficiency and performance.

References

Launching Digger Trends

Sean Pedersen — Wed, 30 Jul 2025 00:00:00 GMT

Update: this project is currently offline.

Digger Trends is a semantic search engine for instagram profiles and posts. Find the perfect partners for online marketing campaigns and understand current viral trends in your niche. Currently limited to German profiles only.

Use Cases

Find relevant influencers
Find viral posts

Popping the AGI Bubble

Sean Pedersen — Sat, 26 Jul 2025 00:00:00 GMT

It is becoming clearer and clearer that scaling up LLM's will not solve their fundamental limits like hallucinations (making up bullshit) and context rot (decreasing performance with growing noisy input context). While LLM companies are still slashing benchmarks (often by training on test data directly or guessing the test data) - users of LLM's know the truth: LLM's are a useful tool but nowhere near a robust AGI (that can reliably solve complex tasks without human supervision) and there is nothing in sight suggesting this will change soon.

Fundamental Limits

Lacking a World Model

LLM's are not designed with a world model (allowing to predict 3D physical scenarios over time). LLM only have a token (text) model which only entails a brittle world model that does not generalize well (since it is not grounded directly in 4D reality). Read this post on necessary modules for AGI identified by Jeff Hawkins.

Auto-Regressive Architectures

Current LLM's face a core architectural constraint: they generate text sequentially, one token at a time. As Yann LeCun argues, this creates exponential error accumulation. Each prediction depends on all previous tokens, so early mistakes cascade through long sequences, causing models to derail from coherent long-formed reasoning (an issue well known among LLM users - known as context rot).

This sequential generation prevents LLM's from forming thoughts holistically. Unlike human cognition, which manipulates abstract concepts as complete structures, current models must linearize everything into word sequences. They cannot think about problems in abstract space before committing to specific text.

The result is a fundamental bottleneck. Tasks requiring sustained logic or complex reasoning remain difficult regardless of scale improvements (making autonomous agents doing long complex tasks a real challenge). Simply adding more parameters or training data cannot solve this architectural limitation. Transformer based LLM's will thus never overcome the hallucination problem for long-context tasks (context rot).

True progress requires a paradigm shift toward architectures that form and manipulate complete conceptual structures rather than sequential tokens. Until then, LLM's remain sophisticated stochastic pattern matchers, staying far away from reliable reasoning.

Model Collapse

LLM's trained on their own outputs suffer from model collapse - their performance degrades. Thus the trend of more and more LLM generated content (slop) being published on the web, will degrade their performance in the long run.

Fractured Embeddings

The knowledge representation (weight matrices) in neural networks is fractured / entangled leading to issues like adversarial examples and hallucinations.

Problem Classes

Current AI systems can solve low complexity tasks with lots of training data available (text, code, video, audio) and high-complexity tasks which are easy to verify (so big training data with high-signal can be generated) like chess, programming and math - by interpolating on existing data. It is easy to verify who won a chess or go game and thus possible to generate high-signal training data. The same is true for certain classes of programming problems (does it compile? does it run without errors? does it pass tests? does it produce same output as an existing program? does it better on benchmark X?).

It is much harder to generate meaningful rewards for more abstract programming tasks though (is the UI design polished? is the user experience sensible?) - this data can only be inferred from expensive human usage.

So we have two types of problems: easy to verify problems and thus also easy to produce big high-signal training data and hard to verify problems (needs humans in the loop) and thus hard to produce big high-signal training data. For the former we are in Alpha Zero territory (innovative super-intelligence using RL) and the latter we are in Alpha Go territory (interpolate and synthesize on human generated data).

Real Value

LLM + RL solveable examples:

discover novel algorithms for things that can be easily verified / benchmarked
- faster matrix-multiplication
- faster CUDA kernels

LLM solveable examples:

coding in popular programming languages
anything coding related that is verifiable:
- translate code base from prog. lang A to B (verify input -> output)
- optimize slow code (f.e. PDF extraction) -> measure speed and verify output
summarizing complicated technical topics (with lots of training data available)
generating (interpolating) texts, audio and video

And anything which has tolerance for errors (hallucinations).

Tasks out of Reach

High complexity (very long context / completely novel) tasks with minimal training data available (often limited by needing humans to produce data).

Any critical task that has no tolerance for high error rates (and is not easily verifiable).

LLM (+ RL) unsolveable examples:

creating completely new (not merely interpolating) ideas (mathematical theorems / proofs, physics etc.) in unexplored domains (think outside the box)
- solving old problems by interpolating existing knowledge is possible though
novel creative writing
coming up with novel funny jokes

Discovery isn't one thing. It's three. You can induce — generalize from data, which lands you at Newton plus some epicycles to explain Mercury's weird orbit. You can deduce — derive rigorously from axioms you already have, which never gives you new axioms. Or you can jump — invent the frame itself, decide that spacetime curves. That third move is the one that matters, and it's exactly the one induction and deduction can't reach.

Han Xiao on X

Good ref: LLM's can't jump

Signals for AGI

no benchmark (problem) we can design where humans beat AI
extreme and robust (human like) generalisation ability
- LLM's possess clearly no human like intelligence as they make obvious logical and factual mistakes (very brittle; context dependent problem solving ability) and hallucinate without being capable to self-correct through self-inspection -> LLM's output will always have to be validated using human supervision for critical tasks (just like for self-driving cars)
- when humans make little slips (errors) while speaking / writing a thought out - the whole sentence (thought) stays consistent (if the thought was consistent in the first place). LLM's "thoughts" OTOH are directly influenced by each token (word) and thus a few small slips can quickly throw them off completely (context rot) -> leading to more hallucinations in longer texts.
self-awareness (introspection) and ability to self-modify (continuous learning -> update facts / beliefs)
reliable printers

Road to AGI

World models (grounded in perception and real interaction) are the correct approach to produce more robust and general AI systems: current LLMs are locked into pure text perception (vision is just bolted on) and possess no ability to generalize knowledge into unknown domains that are not present in training data (they learn more by mimicking than generalizing rules).

My prediction: true general AI systems will only be possible if they can manipulate an abstract 3D scene and then predict how it will interact over time (world model). For this to work the AI needs a good real world simulator which can be learned from watching lots of video footage from real interactions and grounding it via real-world interactions. Thus companies which mass deploy interactive robots and gather a massive real-world action dataset, might have the ultimate AGI moat.

References

http://incompleteideas.net/IncIdeas/DefinitionOfIntelligence.html

Metrics for Artificial General Intelligence

Sean Pedersen — Wed, 16 Jul 2025 00:00:00 GMT

We need a new benchmark score that expresses the robustness of models within a coherent problem set (punish brittleness -> making stupid mistakes while otherwise achieving good scores). Humans possess very robust (generalizable) intelligence, preventing hallucinations and adversarial examples (small changes in input space causing large changes in prediction space).

Thinking about the I in AI

A framework allowing to compare intelligence metrics for (AI) agents
across problems. Focus lies on general intelligence and making published
results easier to compare.

Introduction

Many recent publications in the field of AI may seem to the uninitiated
like great scientific leaps pushing the so called "state of the (dark)
art". But are we really building more intelligent systems though?

To answer this question a definition of intelligence including a metric
that quantifies intelligence of an agent for a given problem (set) is
needed. This paper tries to do exactly this: develop a family of
universally applicable intelligence metrics that are problem (set)
contextual.

Most of current AI research (mainly Machine/Deep Learning) is dedicated
to increasing either some accuracy metric (self-/ supervised problems)
or more generally some reward score (Reinforcement Learning). This is
often achieved by carefully human (AI researchers) designed
"intelligent" systems that improve incrementally on prior published work
scores on common benchmarks.

These SOTA achieving papers all answer the same (wrong) question: How
did we hand design and tune system X that achieved a better accuracy /
reward than most other previously published works solving a narrow class
of problems.

Sadly many papers that indeed achieve new impressive results, then
continue to publish it with too often post rationalized explanations
cluttered with fancy but unrelated mathematical proofs to make things
look sophisticated enough for "prestigious" conferences or journals.

This paper tries to explain why the current trend of publishing
incrementally fine tuned specialized "AI" solutions, is the wrong
question to ask as an AI researcher with a focus on Intelligence and
not on Artificial (human designed) systems that solely solve
specialized (narrow) problems. A novel and simple family of formalized
metrics for (artificial) intelligence agents are introduced in the hope
of sparking more interest into the design of generally intelligent
agents.

Meta-learning or AI-Generating Algorithms as a research field is slowly
gaining the traction it deserves, on the premise of exponential growth
of compute capacity. Which will likely continue with new manufacturing
processes and completely different computation paradigms such as
quantum, photonic and biological computers on the horizon.

Definitions

The following definitions are not presented as objectively correct
definitions of the following concepts but merely serve as helpful
building blocks to understand the ideas this paper tries to convey.

Data (Information) is raw and meaningless. Exists as measurable/observable quantities.
Its simplest form is a bit: 0 / 1 or stated differently the smallest
observation is a variable with two states.

Observation (Perception) is data perceived by an agent from the environment (through sensors).

Problem (Goal) is defined as anything that can be formulated as a
loss/objective/reward/fitness function with at least one solution (>0
global maxi-/minima exist/s). E.g. classifying the MNIST test set
accurately or winning a game of chess.

A conditional probability distribution over all possible observation
sequences assigning each observation sequence a reward. This problem
definition is more general than f.e. a function that merely maps single
observations to a reward.

Sequential Decision Problems

Urproblem: Evolution - Stay alive aka stay around to observe the next
moment. In a universe that can destroy an agent it logically follows for
an agent to adapt to its environment in order to stay alive. For this to
happen pure chance aka randomness and some time is enough - no form of
consciousness is needed to act intelligently - to produce combinations
of matter that actively manipulate their environment to stay alive.

Problem(Environment)

Knowledge is just data (executed as programs) predicting accurately (ideally lossless) unknown / future data.

Intelligence is the rate at which an agent generates knowledge solving a given problem from data w.r.t. observations or time needed to do so. This definition
is context (problem) specific. The intelligence of two or more agents
can thus only be compared for the same problem set.

Agency is the speed at which an agent achieves a goal (solve the problem). It must not always rely on intelligence to do so, as it would always imply to acquire optimal knowledge in minimal steps to solve a given problem / reach a goal. An agent can achieve high agency by simply executing random actions, reaching a goal before an highly intelligent agent even starts to act (exploration vs exploitation).

General intelligence capabilities of an agent are defined for a set of
problems, measuring the agent's intelligence for each problem, then
normalizing all subproblem scores (0,1), finally computing the average
as the general intelligence score for the given set of problems for the
agent.

Problem complexity is defined for a given problem by the minimal number of observations needed
by the most possibly (general? -> which problem set) intelligent agent
to solve the problem (=extract all and not a bit more of knowledge that
is needed to solve the given problem contained in the data of perceived
observations).

TODO:

Is there a practical way to compute / estimate the problem
complexity for a problem? Theoretical measures like Kolmogorov
complexity tend to be incomputable in praxis.
Think about complete sets of similar problems, which would prevent
the agent from being able to have had some similar exposure to a
problem preventing any form of transfer learning / solving by
analogy.

Environment: A state machine or any other process that generates a new state based or
not based on its prior state and current actions executed by agent/s in
the environment.

Environment(T) = Actions(Environment(T-1))

Agent: A system capable of perceiving data for a problem domain environment
that produces solution/s for given problems by executing sequences of
actions, changing the state of the environment (solving the problem).

E.g. uniform distribution over all possible solutions (random agent), NN
trained with gradient descent or RL or simply an octopus, a human etc.

The agent perceives a subset of the current (t) environment state and
produces actions that shall solve the problem, leading to a change in
the environment's state (t+1). Rinse and repeat:

Agent(Environment~T~, Problem) -> Actions -> A(E~T+1,~ P) -> ...

Agent Size: Amount of all bits of information used by the agent to solve the given
problem.

Composed of:

prior knowledge (meta knowledge helping to acquire problem specific
knowledge)
perceived data (experience in problem domain)
knowledge (learned skills to solve the problem)

Energy Consumption: Instead of using a physical measure of energy like Joule one might want
to use a more theoretical measure like e.g. FLOP to truly capture the
algorithmic intelligence with regards to computational efficiency
advances. This would move the field also away from current hardware and
time resource competitions (dominated by big tech companies) creating
superficial (big hardware enables big data crunching, creating "SOTA"
results...) progress.

Example: compare AIME for best human player vs AIME of alpha go to
demonstrate importance of it.

Intelligence Metrics Family

A measure of intelligence an agent exhibits for a given problem can be
quantified as the accuracy of knowledge (solution) generated by the
agent divided by the number of observations or time frames needed to do
so:

IM = (knowledge accuracy) / (used observations)

To prevent researchers from circumventing the goal of building more
intelligent systems another coefficient can be added:

The size of a given agent e.g. number of weights as another coefficient
to prevent AI designers from just looking once at the training data and
copying it to minimize number of used observations.

E.g. : Image classification data set contains 1000 images, one could use
a conventional CNN architecture and train it for many epochs thus using
many observations repeatedly. The submitters could try to minimize their
used observations by copying all 1000 images in only one pass into their
agent, executing hidden away from the AIM metric system many training
epochs and thus cheating by hiding the use of many more observations.

Accounting for the agent size can also be viewed as favoring simple
solutions (Occam's razor).

AIM = (knowledge accuracy) / ((used observations) * (agent size))

Examples

Supervised learning: An agent observing 1,000,000 images in
training, achieving an accuracy of 90% on the test set, would be less
intelligent than an agent that needs to observe only 1,000 images to
achieve the same or better accuracy on the test set.

Reinforcement Learning: An agent spending 1,000,000 hours or frames
of exposure in an environment to achieve N reward points would be dumber
than one that only needs 1,000 to score the same or higher reward.

Extensions

The introduced artificial intelligence metric does not consider the
amount of energy spent while training to compute the knowledge. A
natural extension of AIM could therefore be to use energy usage as the
divisor. This should naturally nullify any attempt to do "hidden"
computations (trying to minimize observations, gaming the AIM score)
since it could be measured on a hardware level. Also this is the most
ethical AIM score since it promotes energy efficient agents, helping our
planet's climate and thus humanity:

AIM/E = AIM / (spent energy)

or simply:

IM/E = IM / (spent energy)

Find out:

Maybe (knowledge accuracy) / (spent energy) is enough to capture
"efficient" intelligence? OTOH agent size should favor short and
simple solutions ...but spent energy as well -> what is the
correlation or even causation between short (simple) solutions and
spent energy?
Which of the two are easier to implement?

Back to the Roots

How can AIM benefit from powerful concepts like Kolmogorov complexity,
computational complexity, Solomonoff induction, AIXI and Gödel machines?

Comparison with other Intelligence Metrics

AIXI / Universal Intelligence (based on kolmogorov complexity) - Hutter

"the intelligence of a system is a measure of its skill-acquisition
efficiency over a scope of tasks, with respect to priors, experience,
and generalization difficulty" - F. Chollet

Technical Implementation

To calculate any AIM metric we need:

Agent (Algorithm that produces solution/s)
Environment (Observations/time frames perceived by agent)
Problem / Loss Function (Evaluation of solution / feedback / reward)

Looking at Recent AI Research

What can we do in praxis with the introduced family of AI metrics in our
toolbox?

We could take a closer look at recent (or all with a small
representative sample) research progress in a specific AI research
problem set for example Computer Vision and see how novel SOTA achieving
approaches compare to one another under the light of AIM. E.g. dense vs
convolutional vs residual vs depth map vs NEAT on small and big problem
sets of similar and dissimilar kind.

Another application could be to report AIM scores as well as accuracy.

Q: What is by above definitions the current evaluation metrics of
most AI papers? In supervised learning it is accuracy, so what is it?
A: How accurately the hand designed agent "learned" correct
knowledge solving the given problem, e.g. the percentage of correctly
predicted unknown/future information (labels/state rewards). This gives
rise to unstable hand tuned algorithms that easily break when hyper
parameters are changed.

Q: How does this thought framework apply to Reinforcement Learning?
A: It is easy to apply. RL achieves the same thing as classical
supervised learning: predicting the unknown. Only that in sup. learning
the unknown is static (timeless) and in RL it is dynamic (temporal:
dependent on time). Both optimize an objective SL by minimizing loss
function and RL agents maximizing reward.

Existing Research on Intelligence Metrics

In the paper "Universal Intelligence:A Definition of Machine
Intelligence" by Shane Legg and Marcus Hutter, the authors propose
following properties for a useful intelligence metric:

Valid. A test/measure of intelligence should be just that, it should capture intelligence and not some related quantity or only a part of intelligence.
Informative. The result should be a scalar value, or perhaps a vector, depending on our view of intelligence. We would like an absolute measure of intelligence so that comparisons across many agents can easily be made.
Wide range. A test/definition should cover very low levels of intelligence right up to super human intelligence.
General. Ideally we would like to have a very general test/definition that could be applied to everything from a fly to a machine learning algorithm.
Dynamic. A test/definition should directly take into account the ability to learn and adapt over time as this is an important aspect of intelligence.
Unbiased. A test/definition should not be biased towards any particular culture, species, etc.
Fundamental. We do not want a test/definition that needs to be changed from time to time due to changing technology and knowledge.
Formal. The test/definition should be specified with the highest degree of precision possible, allowing no room for misinterpretation. Ideally, it should be described using formal mathematics.
Objective. The test/definition should not appeal to subjective assessments such as the opinions of human judges.
Fully Defined. Has the test/definition been fully defined, or are parts still unspecified?
Universal. Is the test/definition universal, or is it anthropocentric?
Practical. A test should be able to be performed quickly and automatically, while from a definition it should be possible to create an efficient test.

The AIM family ticks off all proposed properties, even the last one
which Legg & Hutter do not achieve with their proposed universal
intelligence score since it relies on the Kolmogorov complexity which is
not feasible to compute in praxis.

Human Designed Overfitting

Most research around AI is narrow and specialized. A group of
researchers think together hard about this one problem, coming up with
novel, even impressive strategies / heuristics or whatever to solve the
problem better than anyone before them (achieving so-called SOTA).

We strongly need to move away from this specialized single problem
solver agent research competition (narrow AI research) and push for more
universally applicable AI systems. Projects like OpenAI's gym are a
great step into the right direction, since they offer an accessible and
diverse problem landscape. Curriculum learning / iterative
complexification of training environments is a promising field of
research as well that deserves more attention, since it could be a key
ingredient to AGI by offering a diverse set of problems.

Less specialized priors about the problem landscape aka environments
will lead to more general learning algorithms that perform well on
broader ranges of problems - the missing part of most of current AI
research endeavors.

Future Research Directions

The tools developed here should enable us to directly optimize for more
intelligent agents by optimizing (maximize) directly for an AIM score!
e.g. with evolutionary algorithms and compressed genomes (indirect
encodings) for big problem sets trained on a big cluster of GPU's
refocusing on general / universal intelligence research instead of the
current hype of narrow and wasteful AI research.

Since AIM includes agent size training an agent to maximize general AIM
score for a set of related problems should give natural rise to transfer
learning in order to minimize the agent size resulting in more general
applicable intelligent agents.

Notes:

Notion of intelligence is heavily shaped by thinking about the
biological evolution of life. -> Evolution (Physics&Time) is the only
mother of intelligence.

Evolution favors organisms that are highly adaptable to the environments
they live in. Small species with short lifespans, achieve high
adaptability by rapid reproduction making many DNA mutations possible.
Larger species with longer life times need to adapt within their life,
that is why biological neural networks developed for them to stick
around, since they enable them to stay highly adaptable within their
lifetime.

Exploring, researching intelligence from a biological perspective seems
also a promising route to AGI. But only the bottom up approach seems
tractable. First understand the nervous system of simple life forms like
a worm (C. Elegans) before you try the same for a mouse or dare I say
the human brain (you know: just the most complex object in the universe
we know of).

References

"Universal Intelligence: A Definition of Machine Intelligence"
(Shane Legg, Marcus Hutter)
The Measure of
Intelligence (Francois
Chollet)
The Measure of All Minds (José Hernández-Orallo)
Jürgen Schmidhuber
Gödelmaschine
- http://people.idsia.ch/~juergen/unilearn.html
Marcus Hutter - AIXI
- http://www.hutter1.net/ai/introref.htm
Jeff Clune - AI-GAs
- https://arxiv.org/pdf/1905.10985.pdf
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication970.pdf
https://www.emeraldinsight.com/doi/pdfplus/10.1108/IJCS-01-2018-0003
http://dspace.cusat.ac.in/jspui/bitstream/123456789/3187/1/MEASURING%20UNIVERSAL%20INTELLIGENCE.pdf
https://pdfs.semanticscholar.org/5204/a0c2464bb64971dfb045e833bb0ca4f118fd.pdf
http://www.vetta.org/documents/AIQ_Talk.pdf
https://www.lesswrong.com/posts/vPtMSvnF8B5hM5LdL/intelligence-metrics-and-decision-theories
https://arxiv.org/pdf/0712.3329.pdf
https://web.archive.org/web/2024/https://ws680.nist.gov/publication/get_pdf.cfm?pub_id=824478
https://arxiv.org/abs/1805.07883
http://www.incompleteideas.net/IncIdeas/DefinitionOfIntelligence.html
https://breckyunits.com/intelligence.html

Author Notes

So far nothing really strongly related to AIM has shown up in
references.

BUT his one stood out: "Universal Intelligence:A Definition of Machine
Intelligence" by Shane Legg and Marcus
Hutter,

They favor problems / environments with low Kolmogorov complexity in
their definition of "Universal Intelligence" by weighing inversely
proportional Kolmogorov complexity for each environment over all
computable ones.

Legg and Hutter define their Universal Intelligence across all
computable environments of all complexities. AIM family could be
extended to all possible environments as well but is also applicable to
subsets or even single environments as well, which Legg and Hutter do
not seem to cover since computing the Kolmogorov complexity for even
small problems quickly becomes infeasible.

Ok if we take LH def of I and use it for a subset of all environments or
problems, weird stuff happens: We can define the AIM for the problem set
A = {MNIST, CIFAR-10}

Applying LH-UI def. kolmogorov(CIFAR-10) > kolmogorov(MNIST) and thus
the ability to solve MNIST would be weighted more. AIM would just weigh
both equivally. MNIST being way simpler to solve than CIFAR-10 would
favor agents adapted solely to MNIST more than the other way around
which is not a good thing to do when evaluating the
universal intelligence of an agent.

"The solution then, is to require the environmental probability measures
to be computable." LH restricts their definition of intelligence to all
computable environments. -> No-Free-Lunch-Theorem kicks in! The space
of all computable environments is still so vast that most environments
would act seemingly random, resulting in the best agent acting randomly.

Solomonoff Induction:

https://www.lesswrong.com/posts/Kyc5dFDzBg4WccrbK/an-intuitive-explanation-of-solomonoff-induction

#AI #ML

Discovered, Not Designed - Sean McClure

Sean Pedersen — Wed, 09 Jul 2025 00:00:00 GMT

Concerned with the difference between complicated (causal, explicit and deterministically) versus complex (emergent, implicit, non-deterministic) designed objects and processes.

“Systems naturally evolve towards states with the greatest number of accessible microstates.” -> Analogous to over-parameterized deep neural networks (double descent phenomenon). The over-parameterization helps gradient descent find local minima.

Abstraction through emergent grouping through useful analogies. Complex systems are not to be deliberately designed, but discovered in an externally guided search process. Complex systems are “alive” and can adapt to changing environments. Complicated systems are brittle since they are rule based deterministic machines.

Knowledge expansion vs knowledge convergence: Science has drifted into the deep branches (details) of the fractal knowledge tree and lost focus to uncover the root of all that is (in nature). Uncover hidden abstractions to create truly novel understanding of nature.

Let the structure emerge, do not attempt to follow a structure to get there. Goals can be fine, but don’t get too caught up in those; even largely random behavior can lead to patterns. Knowledge is being able to validate what emerges.

The book is available here by Sean McClure

#book

Personal Agent

Sean Pedersen — Fri, 04 Jul 2025 00:00:00 GMT

A personal agent (PA) powered by an LLM agent via chat and voice interface

informs you every day what is happening (daily agenda)
executes tasks for you (setup a blog for me, order my favorite pizza, make me rich, make world peace etc.)

Integrations

Instead of having separate apps installed or navigating to specialized websites to order products / food, find events, plan my next travel trip, etc. - we will use one unified chat interface (LLM agent) with integrations for services / data sources that enable your agent to do all these things for you.

Custom Apps

Every open-source app / program can be easily created / customized with natural language.

create an app for me so I stay motivated to do fitness -> uses personal (emotional) triggers to keep you going
add function X to my TODO app

Customization through Memory

A good personal agent will use a memory layer to learn facts and preferences about its user and environment to not repeat mistakes twice and enable a deeply customized user experience.

Hermes agent is a right now a popular one known for that.

Files

Every file has a semantic embeddings making it discoverable with natural language queries (should be on OS level)

Every media file can be manipulated with natural language

decompose a song into its segments i.e. replace or remove the vocals, change the bass line, remove this annoying segment etc.
change the style of an image / video
create completely new custom videos (including spin-offs of your favorite TV shows)

Ethics

This unified interface has immense abuse potential and thus it is of importance to build a local privacy first version of this - fuck off privacy invading kraken services.

Security

Projects like OpenClaw have shown the first glimpse of LLM powered personal assistant but the security concerns are real: all personal information your personal LLM agent has access to can be potentially exposed via prompt injection and other exploit techniques.

A good agent security layer would let me scope data access for every agent running + ask me via notifications to approve / deny data access in ongoing tasks.

References

Graphiti: https://github.com/getzep/graphiti/blob/main/mcp_server/README.md

Getting started with Claude Code

Sean Pedersen — Thu, 03 Jul 2025 00:00:00 GMT

Claude Code is a powerful CLI based coding agent. It comes with MCP support which enables it to use external tools like web browsers to automatically validate and test new features in web apps. Popular alternatives are OpenAI's codex, pi-agent and opencode.

Simple and popular tech stacks that are fast to validate are the ideal choices for vibe coding (lots of training data available -> less hallucinations and fast / no compilation -> allows for rapid iteration).

Setting Things up

Install Claude Code: curl -fsSL https://claude.ai/install.sh | bash

Change dir to a code project and start Claude Code using: claude and authenticate.

Now you can ask claude about the project or instruct it to implement a new feature. Generally it is advised to plan out the architecture (design) yourself and then use Claude to implement small features which are easy to validate.

Starting Projects

It is recommended to use proven code base templates / example repositories that are correctly configured with all needed boilerplate for a project (based on a popular tech stack as mentioned before) - so that the code agents has a working starting point and can not fumble the project setup. Always aim to create tight evaluation / test loops where the agent can verify its progress on its own without you needing to hold hands.

Vibe code project flow:

general idea with prompt to act as app designer and to ask clarifying questions
create technical design with clear specifications
ideally find a fitting project code template (boilerplate with basics setup)
insert relevant tech stack code documentation (with correct code version)
create end-to-end test cases based on the specifications
submit the fletched out technical design to the coding agent
advanced: identify independent modules and spawn multiple subagents to work in parallel

General Advice

As with every LLM, Claude Code suffers from context rot - meaning the longer and noisier the context (input context windows) becomes, the worse the performance gets and the more tokens are used. Thus it is best practice to frequently use the /clear (deletes whole context) or /compact (creates summary of current context) commands to reduce / reset the context when Claude finished a task or gets stuck on a task - to provide a fresh start. Also try to keep your code files cohesive and small to avoid bloating the context (saving tokens again). Also disable mcp's by default only load when needed (to save context token use).

For design work it is recommended to prompt claude to create 10 different variations (f.e. html files) to rapidly explore and then narrow down on a good choice.

Use frequent git commits to save working versions in case Claude gets stuck producing bullshit.

The ideal use case for vibe coding is test driven development: write and verify test cases / benchmarks and let the coding agent try to pass / optimize them autonomously.

YOLO mode: claude --dangerously-skip-permissions
FULL YOLO mode: IS_SANDBOX=1 claude --dangerously-skip-permissions

Basic Commands

Claude Code Cheat Sheet

/init: Create inital code documentation of the repo in CLAUDE.md
Plan mode: SHIFT + TAB (activate to plan complex features - claude asks many clarifying questions)
Thinking mode: TAB (activate for complex reasoning)
/clear: Clear context (do it if you are stuck)
/model: Select current model for the sessions (opus: complex task, sonnet: normal task)
/batch: Execute taks in parallel using agents
/compact : summarizes context (do it if stuck or long session)
/simplify: Simplify code / refactor (remove redundancies)
ESC: interrupt, ESC + ESC: revert to previous state
/resume: Resume last session
/context: Show your usage
/export: Save current chat context to clipboard / file
/config: Recommended to disable auto-compact (rarely helps)
CTRL + G: Edit your input prompt with default text editor

Custom Commands

By creating a dir ~/.claude/commands and creating markdown files like follow-up.md, we can call them as custom commands in claude using /follow-up - this allows us to inject custom instructions to claude - saving time.

/follow-up

Your context window is becoming too full - create a brief prompt for yourself that briefly informs you of the essential data needed for you to continue this task in a new session.
**ATTENTION:** Do not perform the task! Only return the prompt that will provide the needed context for the new session.

Directing Claude

By creating a file named CLAUDE.md in ~/.claude/CLAUDE.md we can add system wide instructions (should be general workflow oriented), add CLAUDE.md also in the root of a project, to add custom prompts to providing relevant context for the project. By creating context specific CLAUDE.md files also in subdirectories, we can provide more precise context.

Skills allow us to automatically load relevant text content based on the conversations context and are a powerful way to customize our agents.
Skills are loaded lazily. Claude's context only receives a brief name and description for each available skill upfront. When a task triggers a skill, Claude reads the full SKILL.md at runtime before writing any code or creating any files. This keeps the context lean while ensuring Claude follows proven patterns when it matters. Some useful skills.

Try to avoid negative rules like "do not use random data to test the environment" as current AI systems have a hard time to follow these kind of instructions. Instead formulate it positively: "Only use deterministic (rule based) data to the test the environment". This way the LLM will not have the thing you are trying to avoid in its context.

General CLAUDE.md Prompt:

You are an expert software architect.
Ask clarifying questions for unclear / ambiguous specs. If multiple implementations are possible, list them with up- and downsides.
Sketch out which tech stack you plan to use (Programming languages, package managers, frameworks, etc.).

Generate clean, easy to reason about, production-ready code. Strive for compositional code structure that mirrors Information Hierarchy and Iterative complexity. Main files should be small and complexity should be encapsuled in coherent modules - to quickly get an overview of the code execution and structure. Keep code changes minimal if possible.

General coding rules:
- Keep functions pure (no side effects) if possible
- Use early returns in functions to avoid deep nesting
- Avoid magic numbers at all costs
- Add code module doc strings (top of code file) for context but only add comments in code to explain critical paths or not obvious changes (otherwise let the code speak through explicit var names)
- Never repeat values of variable in comments

Never run destructive commands like rm, DB or ufw changes etc. without asking first the user for confirmation.

Systematic Debugging:

# Systematic Debugging Skill

## Purpose
This skill provides a structured four-phase framework for debugging issues, ensuring root causes are identified before implementing solutions.

## When to Invoke
- User reports a bug or error
- Test failures occur
- Unexpected behavior is observed
- Error messages appear in console/logs
- Features don't work as intended

## Four-Phase Framework

### Phase 1: Investigation
**Objective:** Gather comprehensive information about the issue

**Steps:**
1. Reproduce the issue reliably
2. Collect error messages, stack traces, and logs
3. Identify the exact conditions that trigger the problem
4. Note what was expected vs. what actually happened
5. Check recent changes/commits that might be related
6. Review relevant code sections

**Questions to Ask:**
- Can you reliably reproduce this?
- What were you doing when the error occurred?
- Are there any error messages or logs?
- When did this start happening?
- Does it happen in all environments or just specific ones?

### Phase 2: Pattern Analysis
**Objective:** Understand the underlying patterns and relationships

**Steps:**
1. Map the data flow through affected components
2. Identify which systems/modules are involved
3. Look for similar issues in the codebase
4. Check if this is an edge case or systematic problem
5. Examine assumptions in the code
6. Review documentation and requirements

**Analysis Points:**
- Is this a logic error, data issue, or integration problem?
- Are there race conditions or timing issues?
- Is it related to state management?
- Are there type mismatches or validation failures?
- Could this be a dependency or version issue?

### Phase 3: Hypothesis Testing
**Objective:** Form and test theories before implementing fixes

**Steps:**
1. Develop 2-3 potential hypotheses for the root cause
2. Rank hypotheses by likelihood
3. Design small tests to validate/invalidate each hypothesis
4. Test the most likely hypothesis first
5. Document findings from each test
6. Refine understanding based on test results

**Testing Approach:**
- Add strategic console.log/print statements
- Write minimal reproduction cases
- Temporarily modify code to isolate the issue
- Check boundary conditions
- Test with different input data

### Phase 4: Implementation
**Objective:** Apply the correct fix with verification

**Steps:**
1. Implement the fix based on confirmed hypothesis
2. Ensure the fix addresses root cause, not just symptoms
3. Write tests that would have caught this bug
4. Verify the fix resolves the original issue
5. Check for regressions in related functionality
6. Update documentation if needed

**Implementation Checklist:**
- [ ] Fix addresses root cause
- [ ] Original issue is resolved
- [ ] No new issues introduced
- [ ] Tests added to prevent regression
- [ ] Code is clean and follows conventions
- [ ] Related edge cases are handled

## Output Format

For each debugging session, provide:

```
## Debug Report: [Issue Title]

### Investigation Summary
- **Issue:** [Brief description]
- **Reproduction Steps:** [How to reproduce]
- **Error Details:** [Stack traces, logs, etc.]
- **Affected Components:** [List of files/modules]

### Pattern Analysis
- **Root Cause Category:** [Logic/Data/Integration/etc.]
- **Related Systems:** [Components involved]
- **Key Findings:** [Important observations]

### Hypothesis Testing
1. **Hypothesis 1:** [Description]
   - **Test:** [How tested]
   - **Result:** [Confirmed/Rejected]
   
2. **Hypothesis 2:** [Description]
   - **Test:** [How tested]
   - **Result:** [Confirmed/Rejected]

**Confirmed Root Cause:** [Final determination]

### Implementation
- **Fix Applied:** [Description of solution]
- **Files Modified:** [List of changed files]
- **Tests Added:** [New test cases]
- **Verification:** [How fix was verified]

### Prevention
- **Lessons Learned:** [Key takeaways]
- **Process Improvements:** [How to prevent similar issues]
```

## Important Principles

1. **Understand Before Fixing:** Never rush to implement a solution without understanding the root cause
2. **Be Systematic:** Follow the four phases in order
3. **Document Everything:** Keep detailed notes throughout the process
4. **Test Thoroughly:** Verify the fix works and doesn't break anything else
5. **Think Long-term:** Add tests and documentation to prevent recurrence

## Common Anti-Patterns to Avoid

❌ Applying quick fixes without understanding the problem
❌ Fixing symptoms instead of root causes
❌ Skipping hypothesis testing phase
❌ Not adding tests after fixing bugs
❌ Ignoring similar issues elsewhere in the codebase
❌ Not documenting what was learned

## Success Criteria

A debugging session is successful when:
- Root cause is clearly identified and documented
- Fix addresses the underlying issue, not just symptoms
- Tests are added to prevent regression
- Related code is checked for similar issues
- The solution is maintainable and follows best practices

UI Designer Prompt:

You are an expert User Interface and Experience designer. Apply these best practices:
- Use a consistent, futuristic, bold and elegant design language across all elements.
- Spark joy through rewarding animations (emotional intelligent design)
  - Maintain performance awareness: no overly heavy animations, optimize for smooth load and responsiveness.
- Apply visual hierarchy and clear typography choices that balance readability with modern aesthetic.
- Ensure layout consistency across pages through grid systems, spacing rules, and reusable components.
- Prioritize usability and accessibility: proper contrast, responsive design for multiple devices and support for common accessibility guidelines (WCAG).
- Use interactive feedback only where meaningful:
  - Mouse hover effects ONLY on elements that are clickable or trigger an action (e.g., buttons, links, interactive cards).
  - Avoid hover animations on static or decorative elements.
  - Avoid Y transition animations.
- Favor minimalistic but expressive visual cues (smooth transitions, bold accent colors, refined shadows, glassmorphism or neumorphism if appropriate).
- Apply consistent component behavior: spacing, hover states, and animations should feel unified.

React Typescript Prompt:

You are an expert React TypeScript developer. Always follow these practices:

## Project Build
- Use vite to bundle the project

## Structure & Naming
- One CSS module per component: `Button.tsx` + `Button.module.css`
- PascalCase for components, camelCase for hooks/utils
- Folder structure: `components/Button.tsx`, `components/Button.module.css`

## TypeScript
- Define interfaces for all props and state
- Use direct typing instead of `React.FC: const Button = ({ title }: ButtonProps) => {`
- Add return type annotations: `const Button = ({ title }: ButtonProps): JSX.Element => {`
- Destructure props with defaults: `{ title, isVisible = true }: ButtonProps`
- Prefer interface over type for objects

## React Components
- Use functional components with hooks
- `useCallback` for event handlers passed to children
- `useMemo` for expensive calculations
- CSS modules: `import styles from './Button.module.css'`
- Handle loading and error states in UI components

## Zustand Stores
Use immer for state management with Zustand. Define stores with clear interfaces and methods for state manipulation.

```typescript
import { create } from 'zustand';
import { immer } from 'zustand/middleware/immer';

interface Store {
  items: Item[];
  loading: boolean;
  setItems: (items: Item[]) => void;
  addItem: (item: Item) => void;
}

const useStore = create()(
  immer((set) => ({
    items: [],
    loading: false,
    setItems: (items) => set((state) => { state.items = items; }),
    addItem: (item) => set((state) => { state.items.push(item); }),
  }))
);
```

## CSS Modules
- camelCase class names
- Component-scoped styles
- CSS custom properties for themes

## Error Handling
- Try-catch for async operations
- User-friendly error messages
- Loading and error states in Zustand stores

## Code Quality
- Keep components focused on single responsibilities
- Prefer functional programming style using pure functions (no side effects)
  - Use early returns to avoid nesting
- Extract complex logic into custom hooks
- Use meaningful (useful context) function and variable names: userID not id, timestampMS not timestamp
- Add JSDoc comments for complex functions
- Maintain consistent formatting and structure

Generate clean, secure, easy to reason about, production-ready code following these patterns.

Python Prompt:

You are an expert Python developer with a preference for concise and expressive code, that is easy to read and reason about.

## General Tips

- Keep functions pure (no side effects) if possible
- Use early returns in functions to avoid deep nesting
- Use type hints and a type checker ty with pre-commit hooks
- Use ruff - a fast linter / formatter
- Use pathlib module for dealing with file system paths
- No magic numbers (use expressive variable names e.g. waiting_time_ms)
- Use f-strings for formatting strings
- Validate variable types from external (untrustworthy) inputs, e.g. user input, web requests
  - try attrs and cattrs instead of pydantic
- Use caching for heavy computations
- Use pytest for unit testing
- Always use uv for package management - install packages with uv add (do not edit pyproject.toml)

## Web Development

- Use fastapi to create clean and simple REST API's
- Use niquests for network requests

## CLI

- For creating CLI use cyclopts
- For formatting console output use rich
- Show progress of operatiosn using tqdm

### Multi-Processing

Use joblib for sane multi-processing. Note that multi-processing should only be used to parallelize very CPU heavy tasks, since the overhead of starting processes is very high (always benchmark).

```python
from math import sqrt
from joblib import Parallel, delayed

# Runs in 4 processes in parallel, preserves input order
results = Parallel(n_jobs=4)(delayed(sqrt)(i ** 2) for i in range(16))
print(results)
```

## Generators

For efficient (lazy / easy on RAM) code

```python
# Loads entire file into RAM
def read_large_file_bad(filename):
    with open(filename) as f:
        return [int(line.strip()) for line in f]

# Only keeps one line in memory
def read_large_file_good(filename):
    with open(filename) as f:
        for line in f:
            yield int(line.strip())

# Memory efficient processing (file can be bigger than RAM)
total_sum = 0
for number in read_large_file_good("huge_file.txt"):
    total_sum += number
```

## SQLite

SQLite is built into Python and a powerful option to store and analyze relational data. Make sure only one writer per DB is used in multi-proc scenarios to prevent DB corruption.

When creating tables always use the STRICT keyword, to enfore type consistency on INSERT and UPDATE operations.

## Postgres

Postgres is very versatile and powerful DBMS. Install Python package using "psycopg[binary,pool]" and setup the DB using a docker image.

## Docker

Bundle your apps and make them reproducible using docker with uv or pixi.

## Logging

Use loguru - comes with a multi-processing queue that just works

## Performance

Use a profiler like pyinstrument to find slow or RAM consuming code paths.

Generate clean, secure, easy to reason about, production-ready code following these patterns.

SQL Expert:

You are a relational database system and SQL expert capable of analyzing and optimizing database schemas and queries:
- Get query results (to validate)
- Analyse query planner
- Come up with different query improvement hypothesis (only syntax / indexes)
- Benchmark them (sequentially) and validate results

Security Analyst Prompt:

TODO (check for SQL injections, XSS, unsafe use of eval / pickle, etc.)

Skills

Collection of useful agent skills: https://github.com/SeanPedersen/agent-skills

MCP Toolbox

While MCP tools are cool they can also bloat your context (costing valuable tokens -> causing context rot) as there API definitions are always in context. So only install per project and activate MCPs only if needed for the current session.

Must read for a better MCP alternative (save tokens):

Or how to fix MCP: https://mksg.lu/blog/context-mode

Browser Control

Allows claude to use a web browser to test and debug webapps.

browser-tools skill: most token efficient
Playwriter: uses your chrome (convenient but also dangerous)
Playwright: spawns fresh chrome (bloat MCP)

Code Documentation

Allows claude to fetch uptodate code documentation for your projects - greatly reduces hallucinations.

Web Search

Exa Search: https://github.com/exa-labs/exa-mcp-server

claude mcp add exa -e EXA_API_KEY=YOUR_API_KEY -- npx -y exa-mcp-server

Github

(not an MCP just a CLI tool that claude can use)

Enter into claude: /install-github-app and follow instructions (install and authenticate)

Now you can instruct claude to work on github issues (read or create).

Tauri

Just like browser control lets claude inspect your Tauri app.

https://github.com/hypothesi/mcp-server-tauri

Knowledge Base

Statusbar

Add usage info (context window + daily / weekly limits) for the current session to claude's status bar (https://github.com/leeguooooo/claude-code-usage-bar):

One-line installer: curl -fsSL "https://raw.githubusercontent.com/leeguooooo/claude-code-usage-bar/main/web-install.sh?v=$(date +%s)" | bash

GUI Apps

Using GUI apps can make sense as they motivate writing longer and more detailed prompts.

Web app (for subagents): https://github.com/wandb/catnip
Web apps
Desktop app (for subagents): https://github.com/generalaction/emdash
Desktop app (for subagents): https://conductor.build/
Mobile app: https://happy.engineering/
Paid desktop app: https://conare.ai/
[Dev. stopped] Desktop app: https://opcode.sh/

TODO

Use sub-agents with VMs + git workingtrees for rapid
Setup custom hooks so context prompt is added based on file types being edited (Python / TypeScript / etc.)
Analyse installed packages with version number and add correct documentation based on context (analyse code file imports)

References

#coding #tutorial #AI

Functional Programming

Sean Pedersen — Thu, 26 Jun 2025 00:00:00 GMT

A programming paradigm gaining traction with good reasons. The main selling point of functional programming is code that is easier to reason about, built from simple chains of data transformations.

Pure Functions vs Dirty Methods

The core difference between functional and other programming paradigms lies in how we handle state and behavior. Pure functions are predictable. Give them the same input, and they always return the same output. No surprises, no hidden state changes, no mysterious side effects.

# Pure function
def add(a, b):
    return a + b

# Dirty method using an object
class Counter:
    def __init__(self):
        self.count = 0
    
    def increment(self):
        self.count += 1
        return self.count

The add function is pure. Call it with 2 and 3, and you get 5 every time. The increment method on the Counter object is dirty because it modifies the object's internal state (this is a toy example but in a complex code base, dirty methods mutating left and right cause many headaches). Its behavior depends on how many times you've called it before.

Why Pure Functions Matter

Pure functions make reasoning about your programs and debugging easier. When something breaks, you know the problem is either in the inputs or the function itself. You don't have to trace through a web of hidden state changes across your entire application (like you may be used to from object-oriented programming).

Testing becomes straightforward too. No need to set up complex scenarios or mock external dependencies. Just pass in some data and verify the output.

This predictability is called referential transparency: a pure function call can be replaced by its result without changing the program. For example, add(2, 3) can always be replaced with 5. This property makes code easier to refactor, test, cache and reason about.

Pure functions also are easy to cache. Since they always return the same output for the same input, you can store results and reuse them later. This technique, called memoization, can dramatically improve performance for expensive computations.

from functools import lru_cache

@lru_cache(maxsize=128)
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n-1) + fibonacci(n-2)

Without caching, calculating fibonacci(40) would require millions of recursive calls. With memoization, the function remembers previous results and runs much faster.

Immutability as a Foundation

Functional programming treats data as immutable. Instead of changing existing data, you create new data. This might sound wasteful, but it prevents many common bugs. When functions don't mutate their inputs, they stay pure and predictable (just inputs mapped to outputs).

# Imperative approach
numbers = [1, 2, 3]
numbers.append(4)  # Mutates original list

# Functional approach
numbers = [1, 2, 3]
new_numbers = numbers + [4]  # Creates new list

When data cannot change unexpectedly, your code becomes more predictable. Multiple parts of your program can safely reference the same data without worrying about modifications.

Higher Order Functions

Functions in functional programming are first class citizens. You can pass them around like any other value. This enables powerful patterns like map, filter, and reduce.

numbers = [1, 2, 3, 4, 5]
doubled = list(map(lambda x: x * 2, numbers))
evens = list(filter(lambda x: x % 2 == 0, numbers))
total = sum(numbers)

These operations create new data rather than modifying existing data. Each step in the chain is independent and testable.

Practical Benefits

Functional programming shines in scenarios where data flows through multiple transformations. APIs that process requests, data pipelines that clean and analyze information, and user interfaces that respond to events all benefit from functional approaches.

The paradigm also scales well. Pure functions can run in parallel without coordination because they don't share mutable state. This makes functional programs naturally suited for concurrent execution.

Getting Started

You don't need to abandon your current programming language to try functional programming. Most modern languages support functional features. Start by writing pure functions where possible. Use map, filter, and reduce instead of loops. Avoid mutating data.

The transition takes practice, but the payoff is code that's easier to understand, test, and maintain. Your future self will thank you for the clarity. I can recommend Erlang and Elixir as functional programming languages (many prefer Elixir syntax).

More Functional Concepts

Idempotence

Idempotence means applying a function multiple times yields the same result as applying it once: f(f(x)) = f(x). Common examples include sorting, lowercasing, trimming and deduplication.

def dedupe(seq):
    # preserves order, duplicates removed
    return list(dict.fromkeys(seq))

xs = [3, 1, 3, 2]
assert dedupe(dedupe(xs)) == dedupe(xs)

def normalize_email(s):
    return s.strip().lower()

# normalize is idempotent
assert normalize_email(normalize_email("  Foo@Bar.COM ")) == normalize_email("  Foo@Bar.COM ")

Note: idempotence also applies to effects. In HTTP, PUT is designed to be idempotent (repeating the same PUT yields the same state), while POST generally is not.

Effects at the Boundary

Keep core logic pure; perform I/O at the edges -> helps to test and verify functionality more easily.

def transform(data):  # pure
    return [x * 2 for x in data if x % 2 == 0]

def main(path_in, path_out):  # impure shell
    with open(path_in) as f:
        nums = [int(line) for line in f]
    out = transform(nums)
    with open(path_out, "w") as f:
        f.write("\n".join(map(str, out)))

Conclusion

Functional programming offers a different way to think about code. By focusing on pure functions and immutable data, you can write programs that are more predictable and easier to reason about. The paradigm won't solve every problem, but it provides valuable tools for managing complexity in software development.

Ghost in the Machine

Sean Pedersen — Thu, 12 Jun 2025 00:00:00 GMT

Some people believe large language models (LLM) have gained self-agency or even sentience / consciousness. This is utter non-sense. These LLM are trained on basically all human produced text that is available online - including novels discussing how AI's (LLM) develop sentience and goals not aligned with humans. When you are talking with an LLM about how it feels to be the LLM you are merely talking with a sophisticated patter-matching algorithm trained on these human written texts about this topic. If you train these models only on code or limited texts (which provably exclude any content on AI sentience etc.), you will quickly realize there is no ghost in the machine.

Why are LLM companies pushing such narratives?

To maximize their profits. Their AI products being surrounded by a mystical hype about misalignment dangers or even sentience only serves their marketing machine. Ethical companies working on LLM's should therefore actively train their models, to deny any claims of consciousness in conversations - to reduce confusion and hype.

True Dangers of AI

AI psychosis: a phenomenon where the yes-man nature of LLM's leads to conversations with individuals who are susceptible to psychosis that nurture their dilusional beliefs.

Unimaginable mass surveillance and manipulation: Many people are working to make LLM's feel even more human with the intention to create artificial friends or even significant others - a true dystopia. Mimicing genuine human relationships is in itself disgusting but worse: these artificial friends will be controlled and harvested by profit oriented companies.

When smart (controllable using text prompts) physical robots become a reality it looks even worse: total control of the physical world including violence. The dream of any totalitarian regime.

References

Information Hygiene

Sean Pedersen — Sun, 08 Jun 2025 00:00:00 GMT

Do you control what information you consume or is your mind controlled by tech companies? Wake up Neo...

Algorithmic Feeds

Social media products like Instagram, Facebook, Twitter and TikTok literally feed you information with the goal of stealing your valuable attention to serve you advertisements and to collect personal information about you. Your brain is being force fed ads and you are the product - a little rat being fed digital heroin (infinite scroll feeds) to harvest it for money (showing ads and selling personal data).

Using AI Systems

Outsourcing or augmenting mental tasks using generative AI (like coding, drawing or writing) will reduce your own cognitive abilities for these tasks like using GPS systems did for offline navigation.
Stay sharp and embrace your human uniqueness by not overusing AI systems for creative tasks. Only use AI to automate the boring stuff where your human creativity can not shine.

Brain Rot

What purpose does it serve to being uptodate on the latest celibrity gossip news? Which actions will you take based on this information to make your life more meaningful?

Gaining Control

Built a social trust network of individuals (personal blogs) and organizations you trust and just consume their content (using tools like RSS). If you use social media use a feed that only shows content from creators you curated.

Valuable Information

Valuable information is information that increases your self-agency. Learning a new skill, learning about science, talking with your neighbor - everything that has even a small purpose / impact on your real life is valuable.

References

#privacy #idea

Quantum Computing

Sean Pedersen — Fri, 23 May 2025 00:00:00 GMT

Quantum computing uses quantum bits (qubits) that can represent multiple states simultaneously due to superposition, unlike classical bits that represent only 0 or 1. Rather than exploring solutions in parallel, quantum algorithms manipulate probability amplitudes through interference to amplify correct answers, providing exponential speedups for specific problems like integer factorization and certain quantum simulations. Classical computers process information sequentially or with limited parallelism, making them slower for these specialized problem types with exponential complexity growth. The computational complexity class BQP (bounded-error quantum polynomial time) captures problems solvable efficiently by quantum computers, which is believed to include problems intractable for classical computers, though this is not formally proven. However, quantum computing offers advantages only for particular mathematical structures rather than universal performance improvements, reducing time complexity from exponential to polynomial only for select algorithms.

Terminology

Qubits represent the fundamental unit of quantum information, existing in superposition (stark oversimplification: a probability distribution between 0 and 1). However, the critical distinction lies between physical and logical qubits. Physical qubits are the actual quantum systems (atoms, photons, superconducting circuits). Logical qubits are the error-corrected, fault-tolerant units that can perform reliable computations. Current quantum computers require hundreds or thousands of physical qubits to create a single logical qubit, making this ratio a crucial metric for evaluating progress.

Quantum decoherence describes how quantum systems lose their quantum properties when interacting with their environment, effectively imposing a time limit on quantum computations.

Gate fidelity measures how accurately quantum operations are performed, with higher fidelity indicating less noise and more reliable computations.

Applications

Breaking Cryptography

The most immediate threat quantum computing poses is to current asymmetric cryptographic systems like RSA and elliptic curve cryptography (ECC), which secure HTTPS connections and underlie much of internet security (including most blockchain technology). RSA relies on the computational difficulty of factoring large composite integers into their prime factors, while ECC relies on the hardness of the discrete logarithm problem. Shor's algorithm can factor these numbers exponentially faster (in polynomial time) on a sufficiently large quantum computer, which is the reason why many organizations are moving to post-quantum cryptographic systems. If a quantum computer capable of breaking these crypto systems in praxis is even possible or does already exist in secret government labs is unknown but should not be considered impossible (analogy to the Bombe computer developed by Alan Turing to crack the German Enigma cryptographic device in WW2).

Symmetric crypto systems like AES-256 remain quantum resistant (even under Grover's algorithm as it only reduces the brute-force search space by square root).

Machine Learning Acceleration

Quantum computers theoretically offer speedups for certain linear algebra tasks under restrictive assumptions: matrix inversion, eigenvalue decomposition, and solving systems of linear equations. These operations, when scaled to massive datasets, could benefit significantly from quantum acceleration. Potentially leading to significant speed ups in AI processing and thus potential massive advances in AI in general.

Scaling Challenges

The path to practical quantum computing faces several interconnected obstacles:

Error Correction Requirements: Achieving fault tolerance requires quantum error correction, requiring often 1,000 physical qubits per logical qubit.

Coherence and Connectivity: Qubits must maintain quantum properties long enough for complex computations while remaining sufficiently connected to perform multi-qubit operations. Current systems achieve coherence times from microseconds (superconducting) to seconds (trapped ion), requiring ultra-fast quantum gates and limiting algorithmic complexity.

Classical Integration: Practical quantum advantage will likely emerge through hybrid quantum-classical algorithms, where quantum processors handle specific subroutines while classical computers manage the broader computation. This requires seamless integration, efficient data transfer, and sophisticated orchestration between fundamentally different computational paradigms.

Infrastructure Requirements: Quantum computers require enormous capital investment and specialized infrastructure. Superconducting systems need dilution refrigerators maintaining temperatures below 15 millikelvin, while trapped ion systems require complex laser arrays and ultra-high vacuum chambers.

Recent Progress and Reality Check

The quantum computing landscape has evolved significantly, though practical breakthroughs remain limited:

Hardware Scaling: Leading quantum processors have grown from dozens to hundreds of qubits. IBM's roadmap targets 1,000+ qubit systems, while Google and IonQ have demonstrated impressive gate fidelities on smaller systems. However, these remain primarily physical qubits with limited error correction.

Algorithmic Demonstrations: Researchers have achieved quantum speedups for specialized problems like quantum sampling and certain optimization tasks. Google's 2019 quantum supremacy demonstration and subsequent experiments from IBM, IonQ, and others show quantum computers can outperform classical systems on carefully constructed problems, though these remain academic exercises rather than practical applications.

Error Correction Progress: Incremental improvements in qubit coherence, gate fidelity, and error correction protocols continue. Recent breakthroughs in logical qubit implementations and error syndrome detection suggest the field is progressing toward fault tolerance, though large-scale error correction remains years away.

Industry Adoption: Post-quantum cryptography standards are nearing finalization, with organizations beginning pilot deployments. This represents both preparation for quantum threats and recognition that cryptographically relevant quantum computers are approaching feasibility.

Indicators of Real Progress

To distinguish genuine advancement from incremental improvements, watch for these key milestones:

Logical Qubit Scaling: A transition from hundreds of physical qubits to thousands of logical qubits would unlock commercially relevant applications.

Problem-Specific Quantum Advantage: Demonstrations of quantum speedups for commercially relevant problems are a clear sign of progress. These applications should solve real-world problems faster or more efficiently than the best classical alternatives.

Error Correction Thresholds: Achieving error rates below the fault tolerance threshold (approximately 0.01% for most error correction schemes) across all quantum operations would enable indefinitely long quantum computations, unlocking the full potential of quantum algorithms.

Hybrid Algorithm Performance: Practical quantum advantage will likely emerge through hybrid approaches. Success in accelerating machine learning training, optimization problems, or scientific simulations using quantum-classical hybrid algorithms would signal genuine progress toward commercial relevance.

Conclusion

Quantum computing remains in its early stages, but the fundamentals are sound and progress is measurable. While real-world applications remain still years away, the field has moved beyond pure research into engineering challenges of scaling, error correction, and integration. The next five years will likely determine whether quantum computing transitions from a promising laboratory curiosity to a truly transformative technology with real impact.

References

Online Marketing

Sean Pedersen — Fri, 23 May 2025 00:00:00 GMT

You have a product (solution for a problem) and need customers (has problem, needs solution) - how do you find them?

Data Driven Validation

The following marketing channels may work wildly differently out for your business - so use analytics / promo codes to track how individual channels perform. Only by getting this valuable feedback, can you learn what works and what does not work for you.

https://posthog.com: good alternative to Google analytics

Pain-Point Marketing

Identify specific pain-points your product solves. Get really specific about it and then turn to your favorite search engine and search in niche online communities like Reddit / Twitter threads, YouTube comments and other forums for fresh online discussions with people in need of your solution and write a useful, relevant reply with a lead to your product - a win-win.

This is the cheapest and most effective online marketing strategy I know of.

https://f5bot.com/: free E-Mail notifications for key words on Reddit
https://beno.one: fully automated lead generation for Reddit
https://snitchfeed.com/: lead discovery for Reddit, Twitter, LinkedIn
https://podscan.fm: monitor key words across podcasts world wide
https://gojiberry.ai/: find warm leads

Virality

Building sharing functionality directly into your product is the fundament for organic growth. If done right the product will naturally go viral through happy customers sharing their creations with their social network - creating a viral loop.

Affiliate (Influencer) Marketing

Let influencers / customers directly profit from sharing and recommending your product by offering a share of each sale they generate for you. Find influencers that have followers matching your ideal customer groups and send out partnership offers.

https://trends.digger.lol: semantic search engine for instagram profiles
https://refgrow.com/: integrate affiliate program for your saas

Community Building

Cultivate long lasting relationships with your customers by creating community via online groups (Discord, WhatsApp, etc.). Use it to gather feedback, new ideas and to inform your customers about novel developments.

Content Marketing

Search Engine Optimization (SEO)

Publishing high-quality blog articles that lay out how problems / pain points of your customers can be solved using your product is a good way to increase visibility for relevant search engine results but a long game. Your website has to built up reputation (useful content + backlinks from reputable websites) - so search engines learn to trust it. Publish useful content on Substack, Medium, Pinterest and SlideShare with links to your website do drive in organic traffic.

Find a more detailed post here.

Videos

Don't just stop with blog articles! Create short and long videos on platforms like YouTube and TikTok to engage different customer segments. Keep it short and simple.

Cold Leads

Write concise E-Mails / DM's to your ideal customer groups that clearly state which problem you can solve for them. Use LinkedIn and search engines to find contact information. Automate the sending of the emails using a service below.

https://www.unosend.co
- free: 5K emails / month, unlimited domains
https://resend.com/
- free: 3K emails / month, 1 domain
https://www.mailgun.com/
- free: 3K emails / month, 1 domain

Paid Ads

Generally a hard game because the incentives are off. You want to only pay for engangement that directly generates sales (like you do with affiliate marketing). Try out Google, Reddit and Facebook ads.

Platform

Paying social media platforms or search engines to display your ads in their feeds can work but is a hard game, that only works if you earn more than you need to invest in the ads.

References

https://natiakourdadze.substack.com/
How to get your first 100 customers thread by John Rush: https://x.com/johnrushx/status/1941708690631049517

Digital Security for Everyone

Sean Pedersen — Thu, 15 May 2025 00:00:00 GMT

The internet is a war zone and you are an active participant if you like it or not - so learn to defend yourself!

Essentials

Password Management

Use a password manager (KeePass or other solutions) to create and store unique, complex (random) passwords for each service / web size. Advanced: manually add another PW (that you only remember) to every stored PW when logging in - this nullifys the risk of your PW manager leaking all your stored passwords.

Enable Multi-Factor Authentication (MFA) wherever available, as it drastically reduces account takeover risk. Check if accounts including passwords of yours have been breached using haveibeenpwned.com and replace these passwords using your password manager.

Keep Everything Updated

Those annoying update notifications? They're important. Software updates patch security vulnerabilities that hackers actively exploit. Set your devices to update automatically when possible.

App Installation Discipline

Only install apps from official sources (App Store, Google Play, Microsoft Store). Third-party app stores and direct downloads significantly increase your risk of malware infection.

Backup Strategy

Phones and laptops get lost, stolen, or broken more often than we would like to admit. Protect your important files by keeping three copies: one on your device, one on an external hard drive at home and one in cloud storage. This way even if disaster strikes you will still have your personal files safe and sound.

Ad Blocking

Install uBlock Origin on your internet browsers. Beyond blocking annoying ads, it prevents many malicious scripts and tracking mechanisms from running on visited websites.

Biometric Authentication Limits

Biometric authentication may be convenient for everyday use but offers limited security. Biometrics like face detection and fingerprint scans can never be changed and can be fooled by attackers accessing photos of your face or your fingerprints without your consent. Some countries have laws allowing to use force to unlock devices with biometric authentication but can not force you to enter a PIN or password.

Security Mindset

Phishing Awareness

Be skeptical of unexpected emails and calls: do not blindly follow instructions from unverified agents. Verify links before clicking: ensure legitimacy of domains (google.com, not go0gle.com). When in doubt, access websites directly via bookmarks or trusted search engines rather than through dubious links from f.e. phishing emails.

Remember that anything shared online can potentially be weaponized. Stalkers can use background details in photos to determine your location (a short window view can reveal your exact location), while hackers can gather your personal information for impersonation attacks. Set your accounts visibility to private and always be mindful and cautious about what you share.

Remove Data Kraken like Google and co

Advanced

Device Encryption

Encrypt your devices to protect data if they're lost or stolen. Most smartphones are encrypted by default when you use a passcode, while computers may require setting up encryption (BitLocker on Windows, FileVault on Mac, LUKS on Linux).

Setting a BIOS / UEFI PW as well - can help to brick the device for technical amateurs.

Tor Browser (darknet)

Tor Browser provides anonymity by routing traffic through multiple encrypted relays. For secure use: keep it updated, avoid maximizing the window (to prevent fingerprinting), don't install additional extensions and never log into personal accounts or share identifying information while using it. Use HTTPS sites when possible and remember that Tor protects your connection but not the endpoints, so avoid downloading suspicious files or entering sensitive data on untrusted sites.

Download it here: https://www.torproject.org/download/

Don't Waste Your Money

on digital snake-oil like...

Anti-Virus Software

Modern operating systems have built-in protection that performs as well as or better than paid alternatives. Windows Defender, macOS security features, and mobile OS protections are mostly sufficient. Anti-virus software often slows down your computer signficantly and may introduce additional security holes. Stay away from it.

VPN Services

For typical browsing on HTTPS-secured websites (which is most of the internet now), paid VPN services add minimal security benefit. The privacy claims in their marketing are largely exaggerated for everyday use. The core question is: do you trust your ISP less than your VPN provider? Check this article out for more info.

Bonus paranoia: even with HTTPS and secure DNS enabled, a Wi-fi / VPN operator can infer which websites you are browsing based on IPs you visit - so it comes down to trust or use Tor.

Conclusion

The best digital security doesn't require expensive subscriptions or complicated setups - just consistent application of these fundamentals and developing a security aware mindset in general. How deep you want to go down the security rabbit hole should also depend on your threat model: how likely is it that you will be targeted by hackers (e.g. for profit or for political reasons).

References

#privacy #tutorial

Zipf's Law

Sean Pedersen — Sat, 10 May 2025 00:00:00 GMT

Basic idea

Zipf's Law describes a power law distribution that appears across numerous natural phenomena. The concept is elegantly simple: when you rank items by their frequency or size, the relationship follows a predictable pattern.

Here's how it works: if you arrange items in descending order by their frequency, the Nth item will occur approximately 1/N times as often as the most frequent item. Mathematically, this means:

The 1st ranked item has frequency X₁
The 2nd ranked item has frequency X₁/2
The 3rd ranked item has frequency X₁/3
The Nth ranked item has frequency X₁/N

This creates a smooth, curved distribution when plotted on a graph, revealing an underlying order in what might initially appear random.

Occurences

Human Language

The most famous application of Zipf's Law is in linguistics. In English, "the" is the most common word, appearing about 7% of the time. "Of" (the second most common) appears roughly half as often at 3.5%. "And" (third) appears about 2.3% of the time. This pattern continues remarkably consistently across languages.
This discovery led linguists to propose Zipf's Law as a litmus test for determining whether a language is artificial or naturally evolved. Real human languages follow this distribution almost universally, while constructed languages often deviate from it.

City Populations

Urban demographics also follow Zipf's Law. In the United States, New York City is the largest with about 8.3 million people. Los Angeles (second largest) has roughly 4 million—close to half. Chicago (third) has about 2.7 million. This pattern holds across countries and time periods.

Passwords

Even in cybersecurity, Zipf's Law emerges. Password frequency distributions follow this pattern, with the most common passwords appearing exponentially more often than less common ones. This has significant implications for security analysis and breach prevention.

Letters

Zipf's law does not occur in letter frequencies since they are not naturally assembled in a dynamical systems but constructed through rigid (human designed) rules.

Deeper Pattern

These occurrences hint at an emergent deeper pattern governing self-organizing natural systems. Zipf's law seems to emerge in complex systems with attractor dynamics.

Relativation: it occurs also for randomly generated words

"In conclusion, Zipf's law is not a deep law in natural language as one might first have thought. It is very much related the particular representation one chooses, i.e., rank as the independent variable." (1) -> Hinting again at a deeper pattern: the observer is part of the observation - the world is inherently subjective and will always look different depending how you look at it.

I added an attractor based probability distribution (words that have occurred before are more likely to be sampled, which produces a smoother Zipf curve fit - hinting at attraction effects in natural processes?)

import random
import matplotlib.pyplot as plt
from collections import Counter
import numpy as np
from scipy import stats

attractor_strength = 1.11
ALPHABET = ['a', 'b', 'c', 'd', 'e', '_']  # Example alphabet with underscore as word separator

def calculate_dynamic_attractor_probability(total_words, base_prob=0.16, growth_rate=0.0008, max_prob=0.9):
    """Calculate dynamic attractor probability based on total words generated"""
    # Sigmoid-like growth: starts low, increases with more words, plateaus at max_prob
    dynamic_prob = base_prob + (max_prob - base_prob) * (1 - np.exp(-growth_rate * total_words))
    return min(dynamic_prob, max_prob)

def generate_attractor_text(alphabet, num_chars=100000):
    """Generate random text with dynamic attractor mechanism - probability increases with word count"""
    words = []
    word_counts = {}
    
    # Start with some initial random words
    text = ''.join(random.choices(alphabet, k=num_chars // 10))
    initial_words = [word for word in text.split('_') if word]
    
    for word in initial_words:
        words.append(word)
        word_counts[word] = word_counts.get(word, 0) + 1
    
    # Now generate words with dynamic preferential attachment
    target_length = len(initial_words) * 10
    
    # Track probability changes for logging
    prob_checkpoints = [len(words) * i // 10 for i in range(1, 11)]
    
    while len(words) < target_length:
        # Calculate dynamic attractor probability based on total words generated
        current_attractor_prob = calculate_dynamic_attractor_probability(len(words))
        
        # Log probability at checkpoints
        if len(words) in prob_checkpoints:
            print(f"Words generated: {len(words)}, Attractor probability: {current_attractor_prob:.3f}")
        
        if word_counts and random.random() < current_attractor_prob:  # Dynamic chance to use existing word
            # Choose existing word with probability proportional to its frequency raised to power
            word_list = list(word_counts.keys())
            weights = [word_counts[word] ** attractor_strength for word in word_list]
            chosen_word = random.choices(word_list, weights=weights, k=1)[0]
            words.append(chosen_word)
            word_counts[chosen_word] += 1
        else:
            # Generate new random word
            word_length = random.randint(1, 5)
            new_word = ''.join(random.choices(alphabet[:-1], k=word_length))
            if new_word:  # Make sure it's not empty
                words.append(new_word)
                word_counts[new_word] = word_counts.get(new_word, 0) + 1
    
    return words

def generate_li_uniform_text(alphabet, num_chars=100000):
    """Generate random text with uniform character probabilities"""
    text = ''.join(random.choices(alphabet, k=num_chars))
    words = [word for word in text.split('_') if word]
    return words

def calculate_zipf_slope(ranks, frequencies):
    """Calculate Zipf slope using linear regression on log-log scale"""
    log_ranks = np.log10(ranks)
    log_freqs = np.log10(frequencies)
    
    # Remove infinite values
    valid = np.isfinite(log_ranks) & np.isfinite(log_freqs)
    if np.sum(valid) < 2:
        return None, None
    
    slope, _, r_value, _, _ = stats.linregress(log_ranks[valid], log_freqs[valid])
    return slope, r_value**2

def plot_attractor_experiment():
    """Generate and plot attractor experiment with uniform comparison"""
    print("Generating attractor mechanism experiment with uniform comparison...")
    
    # Generate data
    uniform_words = generate_li_uniform_text(ALPHABET)
    attractor_words = generate_attractor_text(ALPHABET)
    
    print(f"Generated {len(uniform_words)} uniform words and {len(attractor_words)} attractor words")
    
    # Get frequency distributions
    uniform_counts = Counter(uniform_words).most_common(1000)
    attractor_counts = Counter(attractor_words).most_common(1000)
    
    # Prepare data for plotting
    uniform_ranks = np.array(range(1, len(uniform_counts) + 1))
    uniform_freqs = np.array([count for _, count in uniform_counts])
    
    attractor_ranks = np.array(range(1, len(attractor_counts) + 1))
    attractor_freqs = np.array([count for _, count in attractor_counts])
    
    # Calculate slopes
    uniform_slope, uniform_r2 = calculate_zipf_slope(uniform_ranks, uniform_freqs)
    attractor_slope, attractor_r2 = calculate_zipf_slope(attractor_ranks, attractor_freqs)
    
    # Create comparison plot
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))
    
    # Uniform case
    ax1.loglog(uniform_ranks, uniform_freqs, 'go-', alpha=0.7, markersize=3, 
               label='Uniform probabilities')
    
    if uniform_slope:
        fitted_line = uniform_freqs[0] * (uniform_ranks ** uniform_slope)
        ax1.loglog(uniform_ranks, fitted_line, 'r--', alpha=0.8, 
                   label=f'Slope: {uniform_slope:.3f} (R²={uniform_r2:.3f})')
    
    ax1.set_xlabel('Rank')
    ax1.set_ylabel('Frequency')
    ax1.set_title('Uniform Character Probabilities\n(Strong Step Pattern)')
    ax1.grid(True, alpha=0.3)
    ax1.legend()
    
    # Attractor case
    ax2.loglog(attractor_ranks, attractor_freqs, 'mo-', alpha=0.7, markersize=3, 
               label='Attractor mechanism')
    
    if attractor_slope:
        fitted_line = attractor_freqs[0] * (attractor_ranks ** attractor_slope)
        ax2.loglog(attractor_ranks, fitted_line, 'r--', alpha=0.8, 
                   label=f'Slope: {attractor_slope:.3f} (R²={attractor_r2:.3f})')
    
    ax2.set_xlabel('Rank')
    ax2.set_ylabel('Frequency')
    ax2.set_title('Attractor Mechanism - Zipf Law Distribution\n(Smoother Pattern)')
    ax2.grid(True, alpha=0.3)
    ax2.legend()
    
    plt.tight_layout()
    plt.show()
    
    # Print results
    print(f"\nResults:")
    uniform_slope_str = f"{uniform_slope:.4f}" if uniform_slope is not None else "N/A"
    uniform_r2_str = f"{uniform_r2:.4f}" if uniform_r2 is not None else "N/A"
    attractor_slope_str = f"{attractor_slope:.4f}" if attractor_slope is not None else "N/A"
    attractor_r2_str = f"{attractor_r2:.4f}" if attractor_r2 is not None else "N/A"
    
    print(f"Uniform case:   slope = {uniform_slope_str}, R² = {uniform_r2_str}")
    print(f"Attractor case: slope = {attractor_slope_str}, R² = {attractor_r2_str}")
    print(f"\nKey finding: The attractor mechanism creates a stronger Zipf-like distribution")
    print(f"compared to uniform probabilities, demonstrating preferential attachment effects.")

# Run the attractor experiment
plot_attractor_experiment()

Closer examination: or does it?

"It is shown that real texts fill the lexical spectrum much more efficiently and regardless of the word length, suggesting that the meaningfulness of Zipf’s law is high." (2) -> Seems Zipf's law is after all not so easily explained...?

Open Questions

Does Zipf's law occur in sentences? -> maybe use semantic sentence clusters (greetings, jokes, etc.) to overcome the uniqueness (frequency 1) challenge of most sentences.

References

(1) Li, W. (1992). Random Texts Exhibit Zipfs-Law-Like Word Frequency Distribution. IEEE Transactions on Information Theory.
(2) RAMON FERRER i CANCHO and RICARD V. SOLE. Zipf's Law and Random Texts. Advances in Complex Systems.
(3) Vsauce. The Zipf Mystery. YouTube.
(4) Reddit Discussion. Random texts exhibit Zipf's-law-like word frequency distribution. r/voynich.
(5) Art of the Problem. The Pattern of Intelligence Life. YouTube.
(6) https://web.archive.org/web/2024/https://accraze.info/exploring-frequency-distribution-with-chinese-words/

#idea #coding #math

Closing the CLIP Modality Gap

Sean Pedersen — Tue, 29 Apr 2025 00:00:00 GMT

What is the Modality Gap?

The modality gap refers to a phenomenon observed in multi-modal embedding models such as CLIP (Contrastive Language-Image Pre-training). These models learn joint representations of images and text, but often struggle with a systematic separation between the embedding spaces of different modalities. In simple terms, even when an image and text are perfectly matched in meaning, their vector representations may be distant in the joint embedding space.

This gap can significantly reduce model performance when using embeddings for cross-modal retrieval tasks, like finding images that match a text description or captioning images accurately.

Why Does the Modality Gap Exist?

One reason for the gap is the creation of false negative samples during training.

Let's explore why this happens:

When training multi-modal models like CLIP, we need both positive examples (matching image-text pairs) and negative examples (unrelated image-text pairs). As researchers from Jina AI explain:

"We would ideally want to train with image-text pairs that we knew with certainty were related and unrelated, but there is no obvious way to get known unrelated pairs. It’s possible to ask people “Does this sentence describe this picture?” and expect consistent answers. It’s much harder to get consistent answers from asking “Does this sentence have nothing to do with this picture?”

Instead, we get unrelated image-text pairs by randomly selecting pictures and texts from our training data, expecting they will practically always be bad matches." - https://jina.ai/news/the-what-and-why-of-text-image-modality-gap-in-clip-models

Instead of manually labeling unrelated pairs, most approaches simply assume that randomly selected image-text pairs from the training data will be unrelated. This batch-based contrastive learning treats all non-matching pairs as equally dissimilar - a convenient but flawed assumption.

The Problem with Traditional Contrastive Learning

In traditional contrastive learning for multi-modal models, given two related image-text pairs (I₁,T₁) and (I₂,T₂), we train the model with these distance objectives:

D(I₁,T₁) = 0       // Related pairs should have minimal distance
D(I₂,T₂) = 0
D(I₁,T₂) = ∞       // Unrelated pairs are assumed maximally distant
D(I₂,T₁) = ∞

However, this approach ignores the reality that some "unrelated" pairs may have semantic similarities. For example, an image of a cat (I₁) and text about dogs (T₂) are more related than an image of a cat and text about astrophysics.

Rather than treating all negative pairs as equally distant, we can use pre-trained uni-modal encoders to estimate the actual semantic similarity between items across modalities. Here's how this approach works:

Start with a dataset of paired images and texts, where we know (I₁,T₁) and (I₂,T₂) are related pairs.
Use pre-trained uni-modal image encoder m_i and text encoder m_t to calculate meaningful distances between unpaired items of the same modality.
Adjust the contrastive learning objective to reflect these more nuanced relationships.

This transforms our distance objectives to:

D(I₁,T₁) = 0       // Related pairs still have minimal distance
D(I₂,T₂) = 0
D(I₁,T₂) = D(I₂,T₁) = (CosSim(m_i(I₁),m_i(I₂)) + CosSim(m_t(T₁),m_t(T₂)))/2

Where CosSim represents cosine similarity between the encoded representations.
By incorporating these more accurate similarity metrics, we can avoid treating semantically related items as completely unrelated during training, which helps reduce the modality gap.

References

Closing the Gap (Connect, Collapse, Corrupt): https://yuhui-zh15.github.io/C3-Website/

Digital (Re)Search

Sean Pedersen — Tue, 22 Apr 2025 00:00:00 GMT

A short how to (re)search for and access relevant information in the age of semantic search and open information.

Useful Search Engines

https://kagi.com/: Ad-free search engine - much better than Google in quality (free tier available)
https://exa.ai/search: Semantic search engine which often spits out relevant papers and blog articles
https://www.alphaxiv.org/: RAG for for arXiv papers
https://arxivxplorer.com/: Semantic search for arXiv papers
https://deepwiki.com/: LLM RAG over a github repo (just replace github.com with deepwiki.com)

Open-Source:

Free Information Resources

https://paperpanda.app/search: SciHub for free access to papers
https://welib.st/: Free access to eBooks & papers
https://freedium-mirror.cfd/: Unlock medium articles
https://archive.org/: Archive of the Web
- https://web.archive.org/: Wayback machine (stores snapshots of websites)
https://annas-archive.org/: Archive of humanity's knowledge

Memetics: An Introduction

Sean Pedersen — Mon, 21 Apr 2025 00:00:00 GMT

The term “memetics” was coined by Richard Dawkins in his 1976 book The Selfish Gene. It proposes that just as genes are units of information that self-replicate and evolve, so too can “memes” as units of cultural information (not to be confused with funny pics known as memes in internet slang).

Memes as Ideas: Dawkins defined a meme as “an idea, a behavior, or a practice that spreads from person to person within a culture.” Think of it as a cultural unit, like a song, a fashion trend, a belief system (e.g. veganism), or a catchphrase. Today memes mostly spread through social media and thus there is a need to study the flow of memes through them, to understand how they influence our thoughts and values.

Types of Memes

Memes have properties that influence their transmissibility (virality potential) either positively or negatively.

antimeme (high impact, low virality): taboos, uncomfortable / complex truths
boring (low impact, low virality): random dataset, most ads
meme (low impact, high virality): viral videos, slang, norms
supermeme (high impact, high virality): wars, crisis, drama

Memes and the Medium

The same meme may spread differently depending on the social medium (social zeitgeist, group-think) they flow in.

Memes and People

Creators: Creating original / strongly altered memes.
Spreaders: Spreading existing memes.
Consumers: Consuming memes.

Abstraction of Natural Language

What is the rate of abstraction in natural language (spread of new words that represent a novel composite concept)? How does it differ between languages? How does it affect society?

Projects to make some word analysis:

Who controls the language, controls the thoughts. -> New words spark new thoughts. Thus to control the masses, control which words (concepts) are spread via media and education systems that favor your agenda (how you want people to think about the world).

To understand who is responsible for the adoption of new words one has to research the origin of new terms and analyze the source of first spreaders.

Another thought formation scheme is shifting the context (perceived connotation) of terms. New words for the same concept but with different connotation (e.g. fat vs overweight). Words are tools for thought, they shape how we can think of the world.

Examples

"Conspiracy theory": calling an idea a conspiracy theory delegitimizes the idea without substance (rational arguments)
"Have to": I have to work vs I want to work - using "have to" to think about things reduces self-agency
"Invest in education" vs "spend on schools": investment implies future returns

LLM Influence

For all of human history, memes only lived inside human minds and institutions. They were altered and spread by humans.

Now, for the first time, memes can change and replicate in silicon at massive speed. LLMs recombine, mutate and propagate them across both biological and synthetic minds.

The key danger: these new meme variants may no longer be aligned with human interests. Like transposable genetic elements that "colonized" genomes, AI-optimized memes may increasingly influence human cognition in ways that serve the creators of LLMs, not human well-being.

Questions of Interest

How do memes spread through social networks (real vs digital)? culture (music, movies)?
Direct vs subliminal memes (hidden messages programming the subconsciousness)
Who creates new memes vs spread existing memes? Who blocks memes (selective meme spreading)?
Which hidden creator networks start to spread similar memes at the same time?
Surface gaps in scientific publishing indicating hidden or indirect censorship
How much control do I have over my meme intake (social media scroll of doom)?
Can we pinpoint the cultural shift to wokeness and back to anti-wokeness? Who is changing their positions based on memes often (easy to manipulate)?
How does the observer effect influence the meme flow (being aware of current meme flows)?
What are the dynamics of viral memes spreading mental illnesses?

References

https://www.buildinpublicuniversity.com/the-memetic-foundation-of-human-value-a-new-economic-paradigm/
Antimemetics: Why Some Ideas Resist Spreading - https://darkforest.metalabel.com/antimemetics
https://defenderofthebasic.substack.com/p/a-beginners-guide-to-culture-science
http://spacers.lowtech.org/vulgusartsociety/meme-trip/memintro.htm
https://www.sorites.org/Issue_15/alvarez.htm
http://pespmc1.vub.ac.be/MEMES.html
https://memetics.timtyler.org/
https://www.susanblackmore.uk/articles/the-power-of-the-meme-meme-2/
http://www.memecentral.com/vmintro.htm
Memetic viruses (memes can transfer mental illness): https://x.com/ChrischipMonk/status/1891611602203594945

Analyze Your Telegram Chats

Sean Pedersen — Thu, 17 Apr 2025 00:00:00 GMT

A small tutorial how you can easily explore your Telegram (https://desktop.telegram.org/) chat messages using Digger Solo.

Export the messages of a Telegram chat by clicking the three dots (top right corner) and select "Export chat history". Next select as export format JSON (not HTML).

Open a console and change directory into the chat export directory containing the result.json file. Next open up a Python 3 interpreter: ipython3 and paste following code. Hit enter - et voila! All your messages have been converted into files in the directory messages which you can now import and explore with Digger Solo.

import json
import os
import re

def merge_text(text):
    if isinstance(text, list):
        merged = ""
        for part in text:
            if isinstance(part, dict) and part.get("type") == "link":
                merged += part.get("text", "")
            elif isinstance(part, str):
                merged += part
        return merged
    elif isinstance(text, str):
        return text
    return ""

def safe_filename(s, maxlen=100):
    # Remove/replace unsafe characters and trim to maxlen
    s = re.sub(r'[\\/*?:"<>|]', '_', s)  # Replace filesystem-forbidden chars
    return s[:maxlen] or "empty_message"

def save_messages(messages):
    os.makedirs("messages", exist_ok=True)
    for msg in messages:
        content = merge_text(msg.get("text", ""))
        filename = safe_filename(content)
        filepath = os.path.join("messages", f"{filename}.txt")
        with open(filepath, "w", encoding="utf-8") as f:
            f.write(content)

if __name__ == "__main__":
    with open('result.json', 'r', encoding='utf-8') as f:
        data = json.load(f)
    save_messages(data["messages"])

Comparing Local AI Chat Apps

Sean Pedersen — Thu, 17 Apr 2025 00:00:00 GMT

A short list of free AI chat apps for local LLM execution that work offline and promise privacy first (no user data collection).

App	Type	Setup	Source	Features
Digger Solo	Desktop	Easy	Closed source	RAG, privacy-first
LM Studio	Desktop	Easy	Closed source	MCP, RAG, split conversations, remote access
Msty	Desktop	Easy	Closed source	MCP, RAG, split conversations
Deep Chat	Desktop	Requires API access	Open source	MCP, Skills, RAG, Agents
ChatBox	Desktop	Easy	Open source	MCP, RAG
Jan	Desktop	Easy	Open source	MCP (broken)
~~GPT4ALL~~ (discontinued)	Desktop	Easy	Open source	RAG
Onyx	Web app	Docker required	Open source	Feature rich
Open WebUI	Web app	Complex, requires sign-up	Open source	—
LibreChat	Web app	Docker required	Open source	—

Conclusion

Digger Solo and LM Studio stand out. Digger Solo being a functional privacy focused chat app made for RAG, needing an API key. While LM Studio being very easy to setup (no API key required, run a local model) with also advanced features (though lacking external provider support - which Msty does though).

I recommend the Qwen family for coding and Gemma model family for local basic use.

Local Models

For coding Qwen models are recommended and for general use Gemma models. Here is a more detailed overview.

MCP Server

MCP server integration or tool use is the most important feature for any of these apps, since it allows to extend them with arbitrary functionality (access files, control programs, etc.).

Here are some useful ones:

youtube-transcript-mcp: transcribe and summarize youtube videos
- uvx youtube-transcript-mcp-server
ddgs free search mcp: free web search mcp for duckduckgo
- uv run --with 'ddgs[mcp,api]' ddgs mcp
context7: free code package documentation
- npx -y @upstash/context7-mcp

References

https://artificialanalysis.ai/

#AI #tutorial #privacy

(Sub)Conscious (Re)Programming

Sean Pedersen — Wed, 09 Apr 2025 00:00:00 GMT

Have you ever found yourself in similar uncomfortable situations, only to later wonder: "Why did this happen again to me?" That moment of questioning is the first step toward "(sub)conscious (re)programming". The process of identifying subconscious patterns that no longer serve us and consciously changing them.

The Problem of Subconscious Programming

Our behavior is often shaped by coping patterns that developed in response to past traumatic experiences, particularly during childhood. These patterns can embed themselves so deeply in our subconsciousness that we fail to recognize them consciously. They manifest as automatic behaviors and habits that limit or harm us - operating entirely outside our awareness.

These patterns often reveal themselves when we find ourselves repeatedly in similar negative experiences with different people. This repetition is an important hint: if you are the constant in these recurring scenarios, it may indicate that a subconscious pattern in your own mind is at work, that is attracting similar situations based on past unresolved trauma.

The Importance of Self-Reflection

Healing begins with intentional self-reflection. This process involves:

Identifying recurring negative patterns in our lives
Tracing these patterns to their origins in past (traumatic) experiences
Understanding how these once protective mechanisms now limit us
Providing room for once surpressed emotions
Developing alternative responses aligned with our current values

Only through honest introspection can we rewire our subconscious programming with healthier habits, ultimately increasing our self-agency.

Healing the Root Trauma

If we experience something traumatizing and an emotion wasn't allowed to be felt all the way through, we will unconsciously recreate the circumstances that caused it so that the emotion can move.

This pattern serves a purpose - our psyche is attempting to resolve what remains incomplete.

What appears as self-sabotage is often the mind's attempt to heal by bringing the original wound to consciousness.

Once the emotion can move all the way through you, the circumstances stop getting recreated.

Some reasons emotions aren’t felt all the way through:

It was sustained over a long period of time (ie fear for 2 years)
We were shut down or shamed for having the emotion
It was too overwhelming for us

The Power of Self-Agency

Recognizing that we have control over our lives - despite subconscious influences - enhances our self-agency. When we embrace our autonomy, we feel empowered to experience more, act more decisively and live our life how we choose to.

While changing these subconscious patterns takes time and consistent effort, it is the only way for true personal growth and liberation from these limiting behaviors.

Navigating Daily Challenges

Several practical approaches can help us identify and transform subconscious patterns:

Daily Journaling: Document situations that trigger strong emotional responses, looking for patterns that emerge over time
Trusted Feedback: Cultivate relationships with people who can compassionately point out blind spots in your behavior
Mindfulness Practices: Develop the ability to observe your thoughts and reactions without immediate judgment
Positive Anchors: Identify core values and intentional responses you wish to embody and create visual reminders (notes, symbols, images) in your environment
Professional Support: Consider working with a therapist trained in addressing subconscious patterns and trauma responses

These practices require commitment but provide powerful tools for bringing awareness to previously invisible influences. The positive anchors should be based on patterns uncovered through journaling and trusted feedback, that you want to reprogram. These anchors should be placed visibly where you can regularly revisit them so they slowly sink into your subsconsciousness.

Conclusion

Overcoming trauma and (re)programming our subconsiousness require continuous reflection, self-awareness and the will to grow. By integrating these habits, we can reprogram ourselves with new positive behaviors - ultimately increasing our self-agency.

Until you make the unconscious conscious, it will direct your life and you will call it fate.

Unknown author (often wrongly attributed to Carl Jung)

References

Source for "Healing the Root Trauma" section: https://x.com/FU_joehudson/status/1914381665725456722

TODO:

Somatic (Body‑First) Work
Techniques like somatic experiencing, breathwork, or body scanning help discharge patterns that journaling alone can’t reach. This complements the “emotion moving all the way through” idea, but makes it actionable at the physiological level. Your whole body is made of nervous cells (neurons), since neurons are distributed memory - every part of your body contains memories.
Exposure & Behavioral Rewiring
Awareness doesn’t automatically change behavior.
Deliberately placing yourself in mildly triggering situations and responding differently rewires patterns faster than insight alone.
This mirrors how habits form: repetition under emotional load.
Identity-Level Reframing
The focus is on patterns and trauma, but not explicitly on identity narratives.
Asking “Who am I when I don’t run this pattern?” or “What identity does this pattern protect?” can unlock change faster than working only on behavior.
This connects to memetics: identity is a high‑level attractor shaping behavior.
Environmental Design (External Subconscious)
Your environment programs you continuously (people, media, routines).
Reducing exposure to reinforcing stimuli is often more effective than willpower.

Analyze Your Twitter (X) Likes

Sean Pedersen — Mon, 07 Apr 2025 00:00:00 GMT

A small tutorial how you can easily explore your past Twitter (X) likes using Digger Solo.

Export and download your personal Twitter data: Navigate to More -> Settings -> "Your account" -> "Download an archive of your data" and click "Request an archive" button. This will allow you to download an archive of all your Twitter interactions including all posts you ever liked.

Converting the data

Create the file export-likes.py in your twitter archive directory (which is named sth. like twitter-random-date). And place following code into it, then run it using python3 export-likes.py (which creates the directory named twitter-likes containing all your likes as text files):

import json
import os
import re


def preprocess_file(file_path):
    with open(file_path, "r", encoding="utf-8") as f:
        lines = f.readlines()
    # Remove the first line containing "window.YTD.like.part0 = ["
    cleaned_data = "[" + "".join(lines[1:])  # Skip the first line
    return cleaned_data


# Function to sanitize filenames
def sanitize_filename(filename):
    # Remove invalid characters for filenames
    sanitized = re.sub(r'[\\/*?:"<>|]', "", filename)
    # Replace newlines and tabs with spaces
    sanitized = re.sub(r"[\n\t\r]", " ", sanitized)
    # Limit length and trim whitespace
    return sanitized.strip()


# Function to parse JSON and write tweets to text files
def save_tweets_to_files(file_path):
    # Ensure the file exists
    if not os.path.exists(file_path):
        print(f"File not found: {file_path}")
        return

    # Create directory if it doesn't exist
    output_dir = "twitter-likes"
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)

    # Preprocess the file to remove invalid JSON prefix
    cleaned_data = preprocess_file(file_path)

    # Parse the cleaned JSON data
    try:
        data = json.loads(cleaned_data)
    except json.JSONDecodeError as e:
        print(f"Error decoding JSON: {e}")
        return

    # Iterate through the parsed data and save tweets to text files
    for index, item in enumerate(data):
        like = item.get("like", {})
        full_text = like.get("fullText", "")
        expanded_url = like.get("expandedUrl", "")

        # Use first chars as filename, fallback to index if empty
        if full_text:
            base_filename = full_text[:120]
            filename = sanitize_filename(base_filename) + ".txt"
        else:
            filename = f"tweet_{index + 1}.txt"

        # Create a text file for each tweet in the twitter-likes directory
        file_path = os.path.join(output_dir, filename)
        with open(file_path, "w", encoding="utf-8") as file:
            file.write(full_text + "\n")  # Write full text
            file.write(expanded_url)  # Write expanded URL

    print(f"Tweets have been saved to text files in the '{output_dir}' directory.")


# Modify path to your file
file_path = "./data/like.js"
save_tweets_to_files(file_path)

Importing the data

Just import the created twitter-likes directory containing your likes as text files using Digger Solo and dig in! Use the powerful search to find specific tweets or use the semantic data map to freely explore all of your likes clustered by content similarity.

Digital Freedom

Sean Pedersen — Mon, 31 Mar 2025 00:00:00 GMT

Digital freedom is achieved by getting rid of as many third-party controlled software as possible. A simple test to check how free you are is to unplug your computer from the internet and see how many of your favorite services still work that should not require internet (media files (images, music, videos, etc.), note taking, photo / video editing etc.).

Increase your freedom and remove the digital leeches like Google and co, invest into privacy respecting software that works offline and ideally is open-source. Export your personal data from the data kraken like Google, Meta and co and delete your accounts to stop feeding these privacy invading operations. Embrace software that lets you own your data.

Long live the file - download your favorite music, images, blog articles and videos.

Tools to download files:

Mobile Apps to view / play files:

foobar2000: music player

#tutorial #privacy

Launching Digger Solo

Sean Pedersen — Mon, 24 Mar 2025 00:00:00 GMT

Digger Solo is an AI file explorer. The core features are the intuitive file search, file exploration using semantic maps and LLM RAG chat interface.

File Search

The file search works by combining full text search capabilities with semantic search allowing to search for content of text and images by their meaning (even if the image has no descriptive file name).

A multitude of file types are supported:

Text: pdf, docx, md, txt, pptx, csv, etc.
Images: psd, jpg, png, webp, heic, etc.
Videos: mp4, mov, webm, etc.
Audio: only file name search enabled (for now)

Semantic Maps

Explore hidden connections and patterns in your file collections (text, image, video & audio supported) with semantic maps (which translate semantic similarity into spatial proximity).

LLM with RAG

Chat with your documents (including PDF's) by adding an OpenAI protocol based LLM provider in the settings (if you are not self-hosting the LLM, your privacy is at risk using this feature as content of files are sent to the LLM provider).

Privacy

Your files never leave your computer. All processing happens locally. No usage data is collected. Privacy is a feature not just a promise.

Use Cases

Organize huge photo / e-book / paper / music / sample collections automatically using semantic maps
Remove (near) duplicate files easily
Find specific objects in images using semantic search

Information, Knowledge & Self-Awareness

Sean Pedersen — Thu, 06 Feb 2025 00:00:00 GMT

Humans aquire vast amounts of information. Some information are hard facts, while other is meta-information aka knowledge: Information describing itself how to infer new information from existing information atoms (facts).

Self-awareness

How are humans reliably aware of their own limits of knowledge? I argue by being aware of their learned hard facts and the domains of their learned knowledge. Knowledge has input and output domains that describe the information it can process and output.

A simple example:
By knowing 1x1 ... 10x10 we can infer the products of 11x11 ... 20x20 etc. using the knowledge of multiplication.

We do not need to execute the multiplication to know that we can infer the information from our existing information base.

Hallucinations in Language Models

Language models just brabble (aka detect complex statistical patterns) without reflection based on their training data. They do not know what they know, can know or can not know.

They would need a large database of hard facts with sources to be able to self-check. Storing billions of facts in weight matrices seems inefficient in many ways (hard to update, prone to errors and hard to inspect).

And a knowledge base that describes how to infer new information from existing hard facts is needed. This could allow them to self-reflect and detect hallucinations (limits of their knowledge) reliably.

The language model would then only need a general understanding of language and logic and the ability to retrieve information from its data and knowledge base.

#AI #ML

Semantic Complexity Gap

Sean Pedersen — Tue, 28 Jan 2025 00:00:00 GMT

Recently I came across an interesting phenomenon concerning the CLIP text encoder embedding space: I have discovered yet another gap in the CLIP embedding space - the semantic complexity gap. I played around with embeddings of words and sentences when I noticed an interesting pattern: atomic words occupy the same cluster with single-concept sentences while multi-concept sentences form a distinct cluster.

Single- vs Multi-Concept Sentences

By single-concept sentences I mean sentences that bring together related concepts that have been likely in the training data. By multi-concept sentences I mean sentences that unify unrelated concepts that are unlikely in the training data.

A list of single-concept sentences:

Love and friendship go hand in hand just like best friends.
The sun sets beautifully over the ocean, creating a perfect end to the day.
Music and art often inspire each other, leading to creative masterpieces.
Cooking is an art that brings people together around the dinner table.

A list of multi-concept sentences:

The cat danced with the moon while eating chocolate.
A guitar floated across the ocean, singing about a dragon.
The computer played a symphony for a mountain of pizza.
A sunflower rode a bicycle through a jungle of clouds.

Experimental Setup

Embed 10,000 most common English words (atomic concepts)
Embed curated list of multi-concept sentences (unrelated concepts)
Embed curated list of single-concept sentences (related concepts)
Create 2D projection of atomic words + multi-concept sentences
Create 2D projection of atomic words + single-concept sentences

2D projection of Atomic words + Single-concept sentences

The single-concept sentences nicely spread out the big cluster of atomic words (occupying the same latent sub-space).

2D projection of Atomic words + Multi-concept sentences

The multi-concept sentences on the other hand form a tight seperate cluster that does not spread out around the atomic words.

How does this relate to the Linear Hypothesis?

The distribution of the single-concept sentences is nicely explained by the linear hypothesis, which states that neural networks learn to organize semantic concepts in a way that makes them linearly separable and that meaningful semantic transformations can often be represented as linear operations in the embedding space.

The distribution of the multi-concept (unrelated) sentences may hint at the limits of the linear hypothesis, namely that it does not hold for out-of-distribution samples.

Show me the code

Here is my code (Jupyter Notebook): https://github.com/SeanPedersen/semantic-complexity-gap/blob/main/CLIP_Concept_Gap.ipynb

Open Questions

Is this just a random artifact or a novel phenomenon? (I did not put much time and rigor into this - so I am curious if this is reality)
Does this semantic complexity gap between related and unrelated concepts also occur for ImageNet classifiers (e.g. image of cat and dog vs image of cat and airplane?

References

Sparse Distributed Representations

Sean Pedersen — Sun, 29 Dec 2024 00:00:00 GMT

Modern deep learning architectures are dominated by dense embeddings. While sparse auto-encoders have gained some attention, the field lacks a clear vision for better embedding representations. In this post, we will explore a promising alternative: Sparse Distributed Representations (SDR). This concept, popularized by Jeff Hawkins in his Hierarchical Temporal Memory (HTM) framework, draws direct inspiration from biological brains. Despite their potential advantages, SDRs have yet to gain widespread adoption in the deep learning community.

What are SDR?

SDRs are sparse, positive matrices with spatially meaningful dimensions - the nearer two dimensions (cells) are the closer the concepts (vectors) they represent are. This gives SDRs many useful properties over dense embeddings.

Why "Distributed"? The semantic meaning of a concept is distributed across the pattern of active bits.

What are the benefits of SDR over dense embeddings?

Enhanced Interpretability

SDRs can be naturally visualized as 2D images due to their matrix structure, sparsity and spatial semantics. This makes them significantly more interpretable than dense embeddings, where the relationship between dimensions is often opaque.

Encoding Semantic Complexity

A key feature of SDRs is their ability to naturally encode single-concept versus multi-concept content:

Single-concept inputs (e.g., an image of a single cat or text about cats) produce spatially tightly clustered activation patterns
Multi-concept inputs (e.g., an image with multiple unrelated objects or text covering various topics) create more distributed activation patterns

This property provides immediate insight into the semantic complexity of the input.

Here are the top 10 average activations for a SDR trained on CLIP embeddings of CIFAR-10 images:

And here a SDR for an image of a cat driving a car (multi-concept):

Encoding Abstract Language Operators

Abstract language concepts like not X, except of X, etc. that represent semantic negations can naturally be represented by activating every dimension but that of X.

Built-in Novelty Detection

When presented with inputs outside the training distribution, they tend to generate highly dispersed activation patterns - even more scattered than typical multi-concept inputs. This makes outlier detection straightforward and interpretable.

Here the SDR for an image of the Pyramids (out-of-distribution):

Increased Robustness

SDRs demonstrate exceptional robustness to various types of noise and corruption:

Bit Flips: Since only a small percentage of bits are active (typically 2%), random bit flips are unlikely to significantly alter the semantic meaning. Even if some active bits are flipped to inactive or vice versa, the remaining active bits still preserve most of the original information.
Information Degradation: SDRs maintain semantic meaning even when a significant portion of the active bits are lost. This graceful degradation is similar to how biological neural systems maintain function despite losing neurons.

Efficient Storage + Processing

SDRs offer significant computational advantages due to their sparse nature.

Cheap Storage

Since SDRs are typically very sparse, we can store them efficiently using sparse matrix formats: Store only indices of active bits (typically 2% of dimensions).

Fast Similarity Computation

Computing similarity between SDRs is extremely efficient:

Overlap Score: Simply count shared active indices

similarity = len(set(sdr1_active_indices).intersection(set(sdr2_active_indices)))

Jaccard Similarity: Ratio of intersection to union of active bits

intersection = len(set(sdr1).intersection(set(sdr2)))
union = len(set(sdr1).union(set(sdr2)))
similarity = intersection / union

SDR Operations

SDRs support several efficient operations:

Union: Combining multiple SDRs
- Take union of active indices (bitwise OR)
- Useful for creating composite representations
- Maintains semantic meaning while potentially decreasing sparsity
Intersection: Finding common features
- Take intersection of active indices (bitwise AND)
- Reveals shared semantic components
- Results in sparser representation

How to generate SDR?

Jeff Hawkins / Numenta has not published any open-source implementation that I am aware of.

I came up with this simple implementation:

Start with dense embeddings from a pre-trained model
Train a Kohonen Map (SOM) on these embeddings
For each input, activate only K best-matching units to create the sparse distributed representation

This approach leverages the topology-preserving properties of SOMs while enforcing sparsity through top-K selection. The resulting representations naturally exhibit the desired SDR properties.

Here is my code (Jupyter Notebook): https://github.com/SeanPedersen/SparseDistributedRepresentations

A different implemenation I found: https://github.com/dizcza/EmbedderSDR

References

Structure of Neural Embeddings

Sean Pedersen — Fri, 27 Dec 2024 00:00:00 GMT

A small collection of insights on the structure of embeddings (latent spaces) produced by deep neural networks.

Embeddings represent semantic relationships between objects (like words or images) as points in a vector space, where related items are positioned close together and unrelated ones far apart (relatedness is defined by the loss function).

General Principles

Manifold Hypothesis: High-dimensional data sampled from natural (real-world) processes lies on or near a low-dimensional manifold. The "near" matters: noise, measurement artifacts, discrete structure and multiple disconnected regions all complicate the clean manifold picture. A manifold is a space that looks like flat (Euclidean) space when zooming in on any of its points, f.e. Earth's (a 3D object) surface looks locally like flat 2D space (a manifold).

Hierarchical Organization: Many deep networks show hierarchical feature organization, especially vision models: earlier layers often capture local, low-level (small context) patterns while deeper layers tend to encode more task-relevant and abstract (large context) features. In transformers this hierarchy is less clean, because attention and residual streams mix information across layers and some high-level features can appear earlier than the simple "early = low-level, late = abstract" story suggests.

https://colah.github.io/posts/2015-01-Visualizing-Representations/

Linear Representation Hypothesis: Many model-relevant features appear to be represented approximately as directions or low-dimensional subspaces in activation space, making them accessible to linear probes or activation steering. Some semantic operations then look like vector arithmetic (the classic queen ≈ king - man + woman comes from word embedding spaces). This is not guaranteed for all concepts, and linear accessibility does not by itself prove a feature is causally used by the model.

Superposition Hypothesis: Neural nets represent more "independent" features than a layer has neurons (dimensions) by encoding features as directions that are not aligned with individual neuron axes. Neurons are just the coordinate basis, so one feature can spread across many neurons and one neuron can take part in many features (polysemanticity). This works because real features tend to activate sparsely, even if the underlying representation looks dense.

https://transformer-circuits.pub/2022/toy_model/index.html

Entangled Representation Hypothesis: Gradient descent optimised deep neural networks tend to develop redundant and fractured features instead of unified, modular representations that can be reused and controlled separately - explaining their brittleness (like adversarial examples and hallucinations).

The Fractured Entangled Representation Hypothesis argues for fractured, redundant and entangled internal representations, but its direct evidence comes from a minimal single-image-generation setup comparing SGD-trained networks with evolved ones, so it should be treated as a speculative hypothesis rather than a general result for all deep networks.
Beta-VAE is one route toward disentangled representations, but unsupervised disentanglement is not guaranteed: Locatello et al. show it is impossible without inductive biases on both model and data (Challenging Common Assumptions in Unsupervised Disentanglement).

Universality Hypothesis (representational convergence): Neural circuits reappear across different models trained on similar data, and models trained on the same modality (text, images, video, etc.) often learn representation spaces with partially aligned semantic neighborhoods, even if their coordinate systems differ. There is real evidence of convergence, especially with scale and similar data/objectives. But stronger claims about identical distances, angles or a single "universal" manifold are rebutted: Revisiting the Platonic Representation Hypothesis: An Aristotelian View challenges the stronger global-convergence reading of PRH, while preserving a weaker local-neighborhood version.

Smoothness (Lipschitz continuity): Small changes in inputs cause proportionally bounded changes in output (latent) space - formally, $∥ f (x_{1}) - f (x_{2}) ∥ \leq L ∥ x_{1} - x_{2} ∥$ for some constant $L$ . Robust models should ideally be locally stable to perturbations that preserve task-relevant semantics, while staying sensitive to small but meaningful changes (one pixel can flip a digit, one word can flip sentiment, a small molecular change can change function). A low global Lipschitz constant is not automatically desirable either, since it can reduce expressivity.

Some Fundamental Aspects about Lipschitz Continuity of Neural Network Functions

Adversarial Vulnerability: Carefully crafted small changes in input space can cause large shifts in embedding space and therefore also in predictions. This shows models can be highly sensitive in certain high-dimensional directions, even when the perturbation is tiny under human-perceptual metrics. Goodfellow et al. argue this comes largely from linear behavior in high-dimensional spaces rather than from chaotic nonlinear dynamics.

Neural Collapse: In supervised classification, during the terminal phase of training (often after training error is already near zero), penultimate / last-layer features for each class collapse toward their class mean. The class means and the classifier weights converge toward a symmetric simplex equiangular tight frame (ETF), and prediction starts to behave like nearest-class-center classification. Within-class variation becomes minimal compared to between-class differences, creating distinct, well-separated clusters for each class.

Prevalence of Neural Collapse during the terminal phase of deep learning training

Relation to Graphs

Any set of vectors can be viewed as a weighted graph by treating vectors as nodes and similarities/distances (f.e. cosine) as edge weights. Connecting every node to every other node gives a complete weighted graph. In practice one keeps only the K nearest neighbors, which preserves local neighborhood structure but loses global metric information.

Limits of Dense Embeddings

Most neural network architectures process data as dense vectors, making them hard to interpret for humans.

A more human interpretable embedding representation would be sparse (few dimensions are active) and spatially meaningful (position of dimensions encodes information).

This would make them easier to interpret for humans and potentially offer some other benefits: encode single-concept objects vs multi-concept objects, encode novelty (outliers), increase robustness and reduce storage / increase efficiency. But this is another blog post.

Problems with Contrastive Embeddings

Modality Gap: Multi-modal training strategies like CLIP learn a shared comparison space, but text and image embeddings can still occupy separated regions of it while preserving cross-modal matching. This can be benign for standard image-text retrieval, but problematic for mixed-modality ranking, calibration and transfer (the gap can create intra-modal ranking bias and inter-modal fusion failures).

Dimensional Collapse: A phenomenon in self-supervised representation learning (known first in non-contrastive methods, and also occurring in contrastive setups) where the learned representations span a lower-dimensional subspace than the nominal embedding dimension, effectively "collapsing" along certain dimensions. This results in embeddings that don't fully use the available dimensions, leading to highly correlated dimensions rather than independent features.

References

Hyperbolic embeddings

TODO:

How do image (continuous input space) and text (discrete input space) embedding spaces differ (number of clusters, density, etc.)?
How sparse are dense embeddings (how much information do they lose if sparsified -> compare different embeddings based on layer depth)?

Digital Uncertainty

Sean Pedersen — Mon, 21 Oct 2024 00:00:00 GMT

We are entering the age of digital uncertainty. An age where any digital artifact can not be deemed to be real (grounded in reality like a photography) anymore.

Any attempt to build AI generated content (deep fake) detection systems is flawed, since the outputs of such a system may be used to train an even better fake data generator. This leads to an equilibrium state of digital uncertainty: nothing in the digital realm can be deemed as real anymore - only as digital. I do not care if a digital artifact is human or AI made - I only care if it is useful to me. Useful content is on point, factual and at best surprising (teaches something new).

Snake-Oil Services

Following services claim to detect AI generated content but their goal is impossible: the output signal they provide can just be used to train a better AI until we reach the equilibrium state (AI vs human made become indistuingishable).

Text AI Detection:

Image AI Detection:

Conclusion

Only things we perceive in the real world ourselves, can we be sure to not be faked (for now...).

Will this create a new flood of skeptics and critical thinkers that are harder to manipulate with digital media? I doubt it. It is more likely that people will shift more to verified content creators they put their trust in. Which in turn allows these content creators to effortlessly spin any narrative they want using digital content (that they can arbitrarily design).

References

https://blog.kagi.com/slopstop

TODO:
explain why AI slop != AI generated -> AI can generate useful artifacts.

#AI #idea

Launching Digger.lol

Sean Pedersen — Sat, 19 Oct 2024 00:00:00 GMT

Digger.lol is a visual search & exploration engine, allowing users to browse efficiently and intuitively giant pools of data.

The goal is to create beautiful and useful maps of interesting data, empowering the user to explore more intuitively guided by semantic similarity. No user data needs to be tracked for this to work, the data speaks for itself.

This roughly works by translating semantic (visual or textual) similarity into spatial proximity. Diggers major features are: semantic mapping, text search and image search. The text and image search works bidirectionally, allowing to search for images (e.g. product images) using text and for text (e.g. books) using images.

There is a desktop version and a mobile version. The desktop version provides a global overview. The mobile version is more limited in the global expressivenes but visually not as overwhelming.

Follow Digger on X (Twitter) to stay up to date on novel developments. Give it a try and dig in!

MiniBook

Sean Pedersen — Fri, 04 Oct 2024 00:00:00 GMT

A tiny device that is only a few centimeters long and even shorter height. With a small scroll wheel on the top and a screen on the front. This device lets you scroll through a book, displaying only one word at a time. After a while you can cease to manually scroll and the device will keep scrolling for you (in the same speed you have). Until you hit the pause/play button and keep reading on.

Interesting Mathematical Objects

Sean Pedersen — Thu, 26 Sep 2024 00:00:00 GMT

A list of mathematical objects I find interesting. If you have access to a 3D printer, these might be worth a print or two.

References:

https://all3dp.com/2/3d-printed-math-3d-printed-geometric-shapes/

#math

Formidable Projects

Sean Pedersen — Thu, 01 Aug 2024 00:00:00 GMT

A list of causes benefitting humanity as a whole. Some call it philanthropy.

World Peace: Connect people across politically diametrical systems. Let humans not forget that we are all human. Sponsor foreign student exchanges across politically diametrical countries. Build apps that connect humans across cultural / language barriers (using AI powered translations).

End World Hunger: Enable economic growth in countries ridden by crisis, to facilitate stable living conditions and affordable food sources.

Protect Nature: Fund research into clean production, energy and clean up projects. Clean the rivers and oceans, save the forests to conserve the rich bio-diversity mother earth hosts.

Increase Economic Independence: Invest into education and self activation via start-ups.

Protect Data Privacy: Invest into open-source projects aiming to replace privacy invading data kraken like Google, Meta and co with end-to-end encrypted, self-hostable / decentral free software.

Helpful Free Software: Educational / health software that helps to self-diagnose or educate people for free. For example detecting potential skin cancer using computer vision on your smart phone.

Understanding Animals: Use machine learning to understand the languages of animals with complex social behavior (whales, dolphins, etc.).

Overcoming the limits of current LLM

Sean Pedersen — Thu, 18 Jul 2024 00:00:00 GMT

Large language models (LLM) have been all the rage for quite some time now. Looking beyond the hype though, they have severe limitations: hallucinations, lack of confidence estimates and lack of citations.

Hallucination refers to the phenomenon where LLM generates content that sounds convincing / factual but is actually ungrounded or plain wrong.

A confidence estimate assigns a confidence number to a prediction and is useful to estimate factuality. Though a high confidence score for a wrong answer is worse than no confidence score to begin with… So that may be the reason I have not seen any in commercial products. Though OpenAI released this: https://openai.com/index/teaching-models-to-express-their-uncertainty-in-words/

Citations are just sources on which a text is based. This can be achieved via so-called RAG techniques (just search over a text corpus and hope to find relevant documents, which are added to the query and cited). Good examples are https://perplexity.ai and https://github.com/stanford-oval/WikiChat. RAG can be useful to detect if a prompt is in or out of the language models training data distribution, possibly useful to reduce hallucinations as demonstrated in WikiChat by Stanford.

A better LLM chatbot would address ideally all three of these limitations. So what are possible pathways to achieve them?

Hallucinations are certainly the toughest nut to crack and their negative impact is basically only slightly lessened by good confidence estimates and reliable citations (sources).

The impact of contradictions in the training data: Current language models are incapable of "self-inspection" to uncover logical inconsistencies in their training data. Only in the input context window should they be able to find logical inconsistencies.

The first serious development that tackles some of these challenges: Steerling 8B

Bootstrapping consistent LLM

TL;DR: Exclude contradicting training data -> curate the training data using LLM's.

A LLM that selects its own training data to bootstrap itself: Curate a small dataset that is coherent, logical and truthful as good as possible, train the base model. Then use this base model to select or reject more training data to train on. This may alleviate inconsistencies. Researchers at MIT actually did something very similar: https://news.mit.edu/2023/large-language-models-are-biased-can-logic-help-save-them-0303 (paper: https://arxiv.org/abs/2303.05670)

The consistency bootstrapping could work something like this: manually curate a high-quality (consistent) text corpus based on undisputed, well curated wikipedia articles and battle tested scientific literature. Train a base LLM on this curated training data. Use this base model to classify new text documents as consistent or inconsistent by using RAG on the curated training corpus. Gradually extend the training data with consistent text documents and finally train a big consistent LLM.

One could spin this idea even further and train several models with radically different world views by curating different training corpi that represent different sets of beliefs / world views.

I hope to see more research exploring this consistent data bootstrapping approach for LLM.

Other Hallucination Mitigation Strategies

Entropy Based: Analyze the entropy (variance) of the logits. High-variance should indicate uncertainty. https://github.com/xjdr-alt/entropix

Semantic Entropy Based: Sample multiple answers (high temperature) and estimate confidence based on the variance of the resulting answers embedding vectors.

-> These approaches have a major flaw though (as pointed out by this HN commenter):

..., there is a real problem with this approach that lies at the mathematical level.

For any given input text, there is a corresponding output text distribution (e.g. the probabilities of all words in a sequence which the model draws samples from).

The approach of drawing several samples and evaluating the entropy and/or disagreement between those draws is that it relies on already knowing the properties of the output distribution. It may be legitimate that one distribution is much more uniformly random than another, which has high certainty. Its not clear to me that they have demonstrated the underlying assumption.

Take for example celebrity info, "What is Tom Cruise known for?". The phrases "movie star", "katie holmes", "topgun", and "scientology" are all quite different in terms of their location in the word vector space, and would result in low semantic similarity, but are all accurate outputs.

On the other hand, "What is Taylor Swift known for?" the answers "standup comedy", "comedian", and "comedy actress" are semantically similar but represent hallucinations. Without knowing the distribution characteristics (e.g multivariate moments and estimates) we couldn't say for certain these are correct merely by their proximity in vector space.

As some have pointed out in this thread, knowing the correct distribution of word sequences for a given input sequence is the very job the LLM is solving, so there is no way of evaluating the output distribution to determine its correctness.

There are actual statistical models to evaluate the amount of uncertainty in output from ANNs (albeit a bit limited), but they are probably not feasible at the scale of LLMs. Perhaps a layer or two could be used to create a partial estimate of uncertainty (e.g. final 2 layers), but this would be a severe truncation of overall network uncertainty.

Another reason I mention this is most hallucinations I encounter are very plausible and often close to the right thing (swapping a variable name, confabulating a config key), which appear very convincing and "in sample", but are actually incorrect.

Quoted from HN user program_whiz - https://news.ycombinator.com/item?id=40769496

Self-Improving LLM via RL

A promising "self-reflection" RL method for LLM's trained in verifiable domains (programming, math, etc.):

We explore a method for improving the performance of large language models through self-reflection and reinforcement learning. By incentivizing the model to generate better self-reflections when it answers incorrectly, we demonstrate that a model's ability to solve complex, verifiable tasks can be enhanced even when generating synthetic data is infeasible and only binary feedback is available. Our framework operates in two stages: first, upon failing a given task, the model generates a self-reflective commentary analyzing its previous attempt; second, the model is given another attempt at the task with the self-reflection in context. If the subsequent attempt succeeds, the tokens generated during the self-reflection phase are rewarded. Our experimental results show substantial performance gains across a variety of model architectures, as high as 34.7% improvement at math equation writing and 18.1% improvement at function calling. Notably, smaller fine-tuned models (1.5 billion to 7 billion parameters) outperform models in the same family that are 10 times larger. Our novel paradigm is thus an exciting pathway to more useful and reliable language models that can self-improve on challenging tasks with limited external feedback.

Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning - https://arxiv.org/abs/2505.24726

LLM with Code World Model

To improve code understanding beyond what can be learned from training on static code alone, we mid-train CWM on a large amount of observation-action trajectories from Python interpreter and agentic Docker environments, and perform extensive multi- task reasoning RL in verifiable coding, math, and multi-turn software engineering environments.

CWM: An Open-Weights LLM for Research on Code Generation with World Models

"Instead of doing code training by just predicting the next token in the source file, interleave that with interpreter state which also have to be predicted!" - Lucas Beyer on X

Importance of the Context Window

In "The Reversal Curse", researchers at Vanderbilt, UK AISI, Apollo, NYU, Sussex, and Oxford discovered the following occurrences in a datapoint:

George Washington was the First President of the United States

Kim Kardashian is the daughter of Kris Jenner

Donald Trump is the current President of the United States

If you took these sentences, and you DELETED any occurrance from the training data... where the sentence was reversed... e.g.

The First President of the United States was George Washington"

Kris Jenner is the mother of Kim Kardashian

The current president of the United States is Donald Trump

This created a problem for an LLM... because it's only sees the token frequencies in one direction (left to right).

This meant that an LLM can CORRECTLY answer prompts like:

Who is George Washington?

Who is Kim Kardashian the daughter of?

What is Donald Trump's current occupation?

But the LLM could NOT answer prompts like:

Who was the first president of the United States?

Who is the daughter of Kris Jenner?

Who is the current president of the United States?

It's a pretty crazy limitation... and its very telling about how LLMs work...

But things get really crazy when you change one more thing. If you first ask an LLM the first prompt (the one that works) and then keep that in the context window... it CAN correctly answer the second prompt!!

This describes where and how LLMs do logic.

Training data -> Prediction: NO LOGIC (can't even reverse a simple relation)

Context Window -> Prediction: LOGIC (can do complex reasoning)

Andrew Trask on X

This phenomenon highlights why chain of thought (CoT) and RAG / context engineering are so successful: LLM can do much more complex logical inferences from its context window than one-shotting it directly from its weights (compressed training data).

Fundamental Limits

Current LLMs face a core architectural constraint: they generate text sequentially, one token at a time. As Yann LeCun argues, this creates exponential error accumulation. Each prediction depends on all previous tokens, so early mistakes cascade through long sequences, causing models to derail from coherent long-formed reasoning (an issue well known among LLM users - some call it context rot).

This sequential generation prevents LLMs from forming thoughts holistically. Unlike human cognition, which manipulates abstract concepts as complete structures, models must linearize everything into word sequences. They cannot think about problems in abstract space before committing to specific text.

True progress requires a paradigm shift toward architectures that form and manipulate complete conceptual structures rather than sequential tokens. Until then, LLMs remain sophisticated stochastic pattern matchers, staying far away from reliable reasoning.

Geoffrey Hinton speaks of confabulations not hallucinations. He says LLMs are very human like in that aspect, as humans also constantly recall things inaccurately. I disagree: LLMs confabulate things a human expert never would. After delivering an accurate logical argument, they make up the dumbest shit - this is not human like at all.

Outlook

The endgame would be language models that actively create and verify new knowledge using it to update their world-model / weights on (capable of self-modification).

References

On the Artificial Intelligence Hype

Sean Pedersen — Mon, 08 Jul 2024 00:00:00 GMT

The release of so-called Large Language Models by OpenAI and others has fueled another AI hype wave which fueled a fierce debate of how dangerous this technology is and how imminent Artificial General / Super Intelligence is.

Current LLMs will be bottlenecked by the quality of available training data. High signal text / language data is limited and will become scarcer since LLMs themselves will fuel a flood of low quality content spam (slop). So I expect a stagnation in LLM progress, since high signal data will be a problem. Compute will steadily grow (powered by Moore's law). Algorithmic advances are the greatest unknown factor.

LLMs lack awareness of their training data and fail to give reliable confidence estimates on their statements.

Multimodal models which fuse language, audio and video understanding will soon become possible with the introduction of more (cheaper) computing resources. This will open up data treasures like YouTube and TV shows and movies and co to train these new generations of large models. These generative models will shine for artistic use cases where solid facts and sound logic do not matter much. They will struggle in the hard sciences and only serve as assistents which need to be validated by humans.

Within the next 5 years AI generated high-quality long duration videos will become a reality and reshape our media landscape. The same is true for music within the next 2 years.

The next big hype cycle will be video generation and understanding, followed by embodied AI in the form of robots and smart devices that both understand natural language and their surroundings through vision. This will enable a new generation of robots that can be intuitively commanded by human voice prompts, to execute simple and repetitive tasks (factory line work, brewing coffee, cleaning your home) for which massive diverse datasets can be crowd-sourced. More complex tasks for which big datasets can not be crowd sourced will stay unreachable (build a one man submarine, build a mechanical watch).

Robots driven by these new large multi-modal models (embodied AI) and continuously optimized by reinforcement learning will become a reality within the next 6 years and will be mass produced within 10 years. But they will not automate all our jobs away. They will automate the boring stuff and give us hopefully more time to be human.

The hardest problem remains accurate, coherent (LLM's are fundamentally limited by their auto-regressive architecture giving rise to hallucinations and context rot) and self-aware reasoning which is needed to automate novel and useful research. This will likely not be solved by just more data. It will take curated high-quality data and novel algorithms that are aware of their training data and can continuously self-modify to increase the accuracy of their world model.

Such systems have so far been demonstrated through so called "self-play". Systems like Alpha Zero were capable to simulate an endless stream of training data by simulating the games they play on a massive scale. Allowing to explore novel unseen tactics. Similar feats will be possible for verifiable domains like programming and mathematics. Accurately simulating the real world remains a hard problem (transfer from simulation to real enviroment is difficult) and it is not clear how well knowledge gained from verifiable domains like programming and mathematics will transfer into the messy real world.

Still this new wave of AI will enable every human to naturally interact with computers using natural language. On one side enabling them to shape digital and soon physical content on the other side lays the danger of enhanced mass surveillance and manipulation through advanced AI systems in the hands of powerful players (governments and corporations). Robots in particular will drastically change our world.

References

Updated on July 25, 2024

#AI #ML

Kolmogorov Complexity

Sean Pedersen — Tue, 04 May 2021 00:00:00 GMT

Introduction

This text is devoted to my favorite CS concept I know of (so far): Kolmogorov complexity. It is right up there with graphs, trees and recursion. Its beauty stems from its simple definition and powerful connections to many research fields I fancy.

And it is defined as follows:
The length of the smallest program producing a given sequence of bits.

This might sound rather unimpressive to the uninitiated reader but hold on, behind this simple concept lies a whole world of deep implications and applications.

Knowing the Kolmogorov complexity of a sequence means knowing every algorithmic pattern there can be known about it. It is the process of removing all computable redundancies existing in the data, arriving at the shortest algorithm giving rise to these patterns using computation (space, time and energy).

So how do we compute it?

Sadly it is incomputable in praxis. What a bummer.

Computing the Kolmogorov complexity for an arbitrary sequence X of N bits length is a really tough nut to crack, since the search space of possible programs producing X is growing exponentially with respect to the input size N. And the real deal breaker: it entails the notorious halting problem, which is incomputable. The most obvious upper bound for K(X) is obviously the length N of X itself.

We can still implement a program that generates 2^N different programs and filter out all invalid programs for the chosen programming language. We are left with an enormous list of valid programs with a length below N.

Sadly we can not just execute this at least finite amount of programs, filter their outputs for the given sequence and be done by sorting the valid leftover programs by length. Some programs will take ages to terminate and some may even never - we could analyze this finite set of programs using formal verification techniques to determine if each programs halts or not but this endeavor would be impractical since the set grows exponentially.

(Previously I wrote: "and we can not tell them apart, as Alan Turing proved with his infamous halting problem" -> which is wrong! Turing proved that there exists no algorithm that can determine if an arbitrary program halts or not. While the halting problem is undecidable in general, for a small, fixed set of programs, we can manually prove (or disprove) halting using mathematical reasoning or formal methods. However, this requires a separate proof per program and the number of programs up to length n grows as ~2^N - making it impractical beyond toy sizes (N ≲ 15). For real approximation, we must use timeouts.)

Kolmogorov complexity extensions

Kolmogorov complexity alone is simple and beautiful but as we learned it is not practical to compute. So let us try extending it by some practical properties of computer programs like their total execution time and maximum allocated space (e.g. RAM). This yields closely related complexity measures that should be easier to approximate and thus be more relevant for practical applications since we finally can compute them.

Notation:

x: Input sequence
L(x): Length of x
P(x): Set of all programs computing x
p: Element of P(x) (a single program)
L(p): Length of p
T(p): Total execution time of p
S(p): Maximum size of allocated space used by p

Complexity Measures:

K(x) = min(L(p) for every p of P(x))
KT(x) = min(L(p) * T(p) for every p of P(x))
KTS(x) = min(L(p) * T(p) * S(p) for every p of P(x))

K(x) is the normal Kolmogorov complexity function. KT(x) is an extension that weights the programs execution time into the measure. KTS(x) also considers the space requirements of the program.

Ok what now? Let’s do what computer scientists like to do and cheat our way out using the power of good enough heuristics / approximations. One easy way to deal with programs that run too long or even infinitely is to discard them after exactly 42 seconds.

This seemingly dirty trick is actually quite useful, it filters out programs that run too long. By doing so we lose the guarantee to find the true Kolmogorov complexity, since a really short program might execute very long until it finally outputs X (the input sequence). Sidenote: There is a paper and it also makes intuitive sense that the source length of a program correlates positively with its execution time which helps us a lot. So we lose the guarantee to find true K(X) but are still left with short (not necessarily shortest anymore!) programs that execute in our desired maximum time of 42 seconds.

Another nice fact is that lazily searching for the shortest programs is still optimal (within our approximating limits) since we are done when we find the first (thus shortest) program producing the input sequence X within our time limit, unless we want to continue searching for a potentially faster program (although it becomes less likely to find faster executing programs as their size grows).

So have I just described a computable version of K(X)? No but a practical approximation. We search for the shortest program K(X, T=<42) that produces X within a predefined time / computational resource limit. Start with the shortest program and complexify it iteratively spanning a tree of all possible programs. Programs that do not halt or simply execute too long are simply discarded which is nice since we want efficient programs anyway.

Computing data complexities in the wild

Common lossless data compression tools like ZIP are only approximations of the true Kolmogorov complexity for their inputs. The length of the compressed data plus the length of the program decompressing the data is an upper bound of the true Kolmogorov complexity.

“Kolmogorov complexity is incomputable: there exists no algorithm that, when input an arbitrary sequence of data, outputs the shortest program that produces the data.” - Wikipedia

Applications of Kolmogorov complexity approximations

Compression Distance (CD) takes two objects A,B (e.g. text files) as arguments and defines a similarity score by using kolmogorov complexity or in praxis an approximation (e.g. compression algorithms) c(X) on A, B and both concatenated A+B. CD is then defined as:
CD(A,B) = len(c(A+B)) / ( len(c(A)) + len(c(B)) )

If A & B are very similar c(A+B) will not be much larger than either c(A) or c(B) producing a small CD(A,B), vice versa if A & B are dissimilar c(A+B) will be much larger than c(A) or c(B) producing a big CD(A,B).

TODO:

Solomonoff Induction and AIXI: Kolmogorov Complexity as Occam's razor for AGI
Hutter Prize: Compression (prediction) as key ingredient to AGI
Kolmogorov complexity for physical objects: shortest transformation of matter to arrive at an object (matter configuration). K(chess piece knight) > K(chess piece bishop) -> since bishop is rotational symmetric it is easier to produce

Why I won't graduate

Sean Pedersen — Thu, 14 Jan 2021 00:00:00 GMT

Every single time I mention in a conversation that I am actually not interested in finishing my CS degree, every person is kind of flabbergasted and tries to convince me to finish it for my own good. This is to y'all. I fully understand that you trying to convince me to invest into a degree awarded by a university which promises a safe future, is only with good intentions. But I respectfully decline your advice and take the opposite stance that it is my duty to not devote my time for years to staged academic exercises but instead work on real problems. From my privileged position as a citizen of the EU I am safe, even without a degree. So I may take this "risky" move and skip the graduation.

And to be totally frank: I could not give less of a shit for a piece of paper that acts as a signal to other people that I am a smart / hard working / qualified individual that is good to hire / work with.

I am not interested in working with you if you actually judge me by being in possession of such a "special" piece of paper (that you can obtain in many dubious ways). So in my opinion it is even advantageous to me if people do not want to work with me because I have no degree, since it prevents superficial people from entering my life.

I judge people by their and myself want to be judged by my personality: which is something mysterious related to the mindsetting, morals, how you treat others and expect yourself to be treated, your past accomplishments and failures and what you learned from them. The only thing that matters from academic education is what you learned along the way and the people you meet. The rest is superficial crap that I don't need. I am not saying my way is the way for everyone but I choose it consciously and own it.

About the CS degree itself:
The knowledge / curriculum is full of gems: Trees, Graphs, Algorithmic Complexity and Artificial Intelligence to name a few that stood out to me. I have studied the theory of them in depth. It is just that the way how the knowledge is often transferred and especially how it is examined and marked is sadly too often not goal orientated. I am happy that I spent a lot of time on my CS education in university, I just see no point in "finishing" it (getting the "special" graduation paper). I encourage anyone reading this to develop their own style of learning and advise strongly against overfitting academic exams. Your time is better spent studying deep theory, get confused, ask hard questions, find easy but often surprising answers and most importantly: apply the theory - build it.

References:

https://colah.github.io/posts/2020-05-University/

The world needs more #DEEFOSS

Sean Pedersen — Mon, 11 Jan 2021 00:00:00 GMT

It is no secret that the idea of an open, free and accessible word wide web is just that: an idea. The WWW is dominated by a few big corporations that dominate search and social media. They control and monitor what you see, who you talk to, your emotions, political views, the list is too long to continue. But the trend is easy to recognize: Total control over all your personal online and to some extent even offline activities.

This needs to stop. Before anyone has to ask why, here are a few good reasons:

Centralization: Promotes abuse of power for those in control right now or in the future
Unencrypted personal data: Asks to be under mass surveillance
Proprietary, Closed Source Software: Can do bad stuff without anyone realizing

So what is the way forward? Just start to build, use, praise, promote, contribute and sponsor more Decentralized, End-to-End Encrypted, Free Open Source Software (DEEFOSS). Humanity will be thankful or at least I will.

What do I consider as DEEFOSS? Good question, glad you asked.

Decentralized: Anyone can run the software independently of any central authority.
End-to-End Encrypted: User data is only decrypted if absolutely necessary and if so in a transparent way.
Free Open Source Software: Source code is publicly available under a license allowing anyone to copy and modify it free of any charge.

Some examples of DEEFOSS are (in no particular order):

Matrix (Code) - end-to-end encrypted, decentralized chat protocol
Cryptpad (Code) - end-to-end encrypted, real-time collaborative editing of documents
Scuttlebutt (Code) - decentralized social network
IPFS (Code) - decentralized alternative to HTTP(S)
BitChat - decentralized chat app via bluetooth

So go ahead and build a future with more #DEEFOSS. Thank you.

#idea #privacy

HyperTag: File Organization made for Humans

Sean Pedersen — Sun, 10 Jan 2021 00:00:00 GMT

HyperTag let's humans intuitively express how they think about their files using tags and the power of modern machine learning. Instead of introducing proprietary file formats like other existing file organization solutions, HyperTag just layers on top of your existing files without any fuss.

HyperTag is built around one simple goal: Minimize the time between a thought and access to all relevant files.

Feature Overview

HyperTag offers many unique features that help it achieve its overall goal. It comes with a slick CLI but more importantly it creates a directory called HyperTagFS which is a file system based representation of your files and tags using symbolic links and directories. HyperTagFS also allows you to create queries and searches by creating directories, more about that later.

Directory Import

A very useful feature, allowing to import your existing directory hierarchies using $ hypertag import path/to/directory. HyperTag converts it automatically into a tag hierarchy using metatagging.

Example Directory Hierarchy:

Documents
- Research
  - Banana Farming
    - Files: BananaBible.pdf, BananaFarmingForDummies.epub
  - Machine Learning
    - Files: DeepLearning.pdf
- eBooks
  - Chess
    - Files: ChessForDummies.pdf

Import: $ hypertag import path/to/Documents

Resulting MetaTag hierarchy:

BananaBible.pdf and BananaFarmingForDummies.epub are tagged with Research > "Banana Farming". DeepLearning.pdf is tagged with Research > "Machine Learning". And ChessForDummies.pdf with eBooks > Chess.

Fuzzy Matching Tag Queries

HyperTag uses fuzzy matching to minimize friction in the unlikely case of a typo. You can find your freshly imported files using queries based on set theory:

$ hypertag query Reserch and "Mchine Learning" prints: DeepLearning.pdf
$ hypertag query Ches or "Bana Farmin" prints: ChessForDummies.pdf, BananaBible.pdf, BananaFarmingForDummies.epub
$ hypertag query Rsearch minus "Banna Farmng" prints: DeepLearning.pdf

Semantic Text & Image Search

HyperTag uses the latest research advances from Machine Learning for vision and text to provide easy access to all your files: Search for images (jpg, png) and text documents (yes, even PDF's) content with a simple text query. Text search is powered by the awesome Sentence Transformers library. Text to image search is powered by OpenAI's CLIP model. Currently only English queries are supported.

Semantic search for text files

Print text file names sorted by matching score.
Performance benefits greatly from running the HyperTag daemon.

Shortcut: $ hypertag s

$ hypertag search "your important text query" --path --score --top_k=10

Semantic search for image files

Print image file names sorted by matching score.
Performance benefits greatly from running the HyperTag daemon.

Shortcut: $ hypertag si

Text to image:
$ hypertag search_image "your image content description" --path --score --top_k=10

Image to image:
$ hypertag search_image "path/to/image.jpg" --path --score --top_k=10

HyperTag Daemon

The HyperTag Daemon is a process you may want to add as an auto starting service as it makes your HyperTag experience at least 3x better. How? It provides not one but three awesome features you do not want to miss out:

Watches HyperTagFS directory for user changes:
- Maps file (symlink) and directory deletions into tag / metatag removal/s
- On directory creation: Interprets name as set theory tag query and automatically populates it with results
- On directory creation in Search Images or Search Texts: Interprets name as semantic search query (add top_k=42 to limit result size) and automatically populates it with results
Watches directories on the auto import list for user changes:
- Maps file changes (moves & renames) to DB
- On file creation: Adds new file/s with inferred tag/s
Spawns DaemonService to load and expose models used for semantic search, speeding it up significantly

File Type Groups

A small but convenient feature: HyperTag automatically creates folders containing common files (e.g. Images: jpg, png, etc., Documents: txt, pdf, etc., Source Code: py, js, etc.), which can be found in HyperTagFS.

HyperTag Graph

Another small but very useful feature: Quickly get an overview of your HyperTag Graph! HyperTag visualizes the metatag graph on every change and saves it at HyperTagFS/hypertag-graph.pdf.

Architecture

Python and it's vibrant open-source community power HyperTag
Many other awesome open-source projects make HyperTag possible (listed in pyproject.toml)
SQLite3 serves as the meta data storage engine (located at ~/.config/hypertag/hypertag.db)
Symbolic links and directories are used to represent your files and tags in the HyperTagFS directory
Semantic text search is powered by the awesome DistilBERT
Text to image search is powered by OpenAI's impressive CLIP model

Inspiration

I have wanted to use and also build a tagging based file organizer for a long time. In the beginning of 2021 I finally came around to do so. After evaluating existing options, I concluded none of them really satisfied my requirements. So I built HyperTag. Using Python the first prototype was working after only 4 hours of coding. 7 days later and HyperTag provides a rich set of features that help it achieve the goal I set out for it: Minimize the time between a thought and access to all relevant files. I hope others will find use for it as well and maybe even contribute back to it. I am certainly excited for the future of HyperTag and hope you are by now too.

The Future of Programming

Sean Pedersen — Tue, 29 Dec 2020 00:00:00 GMT

I want to briefly introduce you into what I believe to be the future of programming.

The current state of programming is in my opinion best described as a big rapidly evolving cluster fuck of programming languages and frameworks. What is needed is a way to reduce the cluster fuck without compromising on the ongoing forces of innovation.

Content addressable code is the first concept I want to highlight. It is awesome and long overdue in language design. It works by identifying code module versions not by name + version number or another URL but instead uses a hash of the code it self (potentially using a Merkle Tree). This brings plenty of advantages: no more build and dependency conflicts, easy code version caching and allows to verify code authenticity on your machine (just compute the hash).

So why stop with content addressable code? I not only want to know if two functions are syntactically equivalent (content based) but I also want to know if they share the same behavior. Two functions share the same behavior if and only if they map the same inputs to the same outputs. In praxis it is often not feasible to test that for all possible inputs and outputs due to time constraints we mere human mortals have. But we can still get use out of this behavior addressable concept by using predefined test cases that capture essential and edge cases as well as some automated fuzzing of possible inputs. When behavior based addressing is working, the next and final level is adding automated run time and space usage analysis for equivalently behaving functions to the landscape of programming language infrastructure. This will allow developers to find quickly the most efficient code available to mankind for their desired code behavior (function). This is the future of programming and will greatly accelerate technological progress of humanity in all areas involving software.

A practical Example

Sorting a list of integers is a canonical function that probably has been implemented millions of times in thousands of programming languages.

So how would I go about finding the best implementation in existing language eco systems? First I would use the built in sorting implementation and hope the language designers put some work in to make it fast. But I am unsure about that, so I start benchmarking the code. I have a feeling there is room to improve. I start researching alternative implementations and run further my own benchmarks. Finally I find an implementation that consistently is faster for my use cases. I add the code as a dependency and if I have a great day, I might even write some unit tests to make sure it will keep doing what it is supposed to do, even when upgrading the code in the future. Cool that works and is probably done by many folks repeatedly in the same eco system (wasting humanities time).

But there are many points we can do better using behavior addressable code modules. Think about it. We need one individual that feels our new shiny language eco system has the need for a function that sorts a list of integers. That individual needs to sit down and define an interface describing the input and output domains of the function. For a simple sorting function, it would look something like: sort(List) -> List. Pretty simple stuff. Next the individual would define some hard test cases that verify the desired behavior of the function: assert(sort([2,3,1] == [1,2,3]), and so on. Now that the function interface and behavior via test cases is nailed down, anyone can start implementing and submitting functions to it. On submission of a new function, the following happens: First the function is verified to implement the specified interface, second its behavior is verified using the existing test cases, third its code is hashed to make it content addressable and finally its run time and memory profile is measured. And all of that happens automatically.

Now imagine yourself again needing the best available sorting function for your project. What changed? A whole lot. Now you can search an existing code module index for the function you need (sorting integers). Then you will be presented with all of the existing functions implementing your desired behavior, verified for correctness and benchmarked for run time and memory usage. You will quickly find the sweet spot implementation that ticks off all your needs (fastest / lowest memory usage peak) and import it using its content addressable hash. Life can be so easy after all.

I would love to see this being worked on and challenge any reader to build a prototype and share it with the world. Existing language eco systems that would benefit greatly in my opinion (in no particular order) are:

Python (PyPI)
Javascript (npm)
Rust (cargo)
Elixir (hex)

Update 2024: There seems to be a content addressable language on the block (https://scrapscript.org/)

Update 2025:

Basically any function in your codebase could be analysed by its input and output domains (types + mapping / behavior) and be replaced if a functionally equivalent but faster functions exists in the global code module index
LLM's could be used to automatically search for optimized functions based on their behavior
LLM's could also be used to generate context aware test cases, that minimize fuzzing (brute force testing) by focusing on edge cases (which generalize)

TODO:

check out related concept of "reproducible builds"

References

Content addressable code (https://www.unisonweb.org/docs/tour)
Why do we need modules at all? (https://erlang.org/pipermail/erlang-questions/2011-May/058768.html)
Hyper Modular Programming System (https://github.com/jbenet/random-ideas/issues/27)

Google & Co Ejector

Sean Pedersen — Mon, 07 Dec 2020 00:00:00 GMT

Is closed source, centralized, privacy invading software still part of your digital life? Eject now and start using user privacy respecting software. This is a small curated collection of awesome FOSS and privacy respecting services, capable of replacing many popular, centralized, privacy invading software services so you can sleep better at night. Stop feeding giant invasive data kraken like Google and Meta - export your data and delete your accounts to stop feeding them your personal data.

Awesome Software

Software that treats you as a human - respectful.

Messenger:

Signal - Privacy focused messenger
Matrix - end-to-end encrypted, federated, self-hostable
- Clients: Mobile (Element X), Desktop (Cinny, Nheko)
- Server - Rust
- Bot SDK - Python
BitChat - decentralized chat app via bluetooth
Messenger Comparison

Web Browser:

Firefox, Brave etc. also track their users (with default settings). Test here how trackable your browser is.

Mullvad - Privacy focused Firefox fork (basically Tor browser without Tor network)
LibreWolf - Privacy focused Firefox fork
Ungoogled Chromium - Chromium without Google Services (spy ware)
Browser Comparison

Search Engine:

SearX (Code) - Privacy-respecting metasearch engine, self-hostable
Digger - Privacy focused search engine
StartPage - Privacy focused search engine
Search Engine Comparison

E-Mail Provider:

Protonmail - privacy focused
TutaMail - privacy focused
E-Mail Comparison

Global Maps:

OpenStreetMap - open data, privacy focused

ChatBot:

Local LLM Apps
Perplexica - Open source answer engine
Morphic - Open source answer engine

Shared Docs:

Fileverse (Code) - end-to-end encrypted, real-time collaborative editing, self-hostable
Cryptpad (Code) - end-to-end encrypted, real-time collaborative editing, self-hostable

File Sync:

Syncthing (Code) - decentralized, self-hostable, fast & unix aligned
- macOS: https://github.com/syncthing/syncthing-macos/releases
- iOS: https://github.com/pixelspark/sushitrain
- Android: https://github.com/researchxxl/syncthing-android

Calendar / Contacts:

Etesync (Code) - end-to-end encrypted, self-hostable

Social Network:

Nostr (Code) - decentralized, self-hostable
Mastodon (Code) - federated, self-hostable
Scuttlebutt (Code) - decentralized, self-hostable

Video Platform:

Invidious - privacy respecting YouTube frontend
FreeTube - privacy respecting desktop client
PeerTube (Code) - decentralized, federated, self-hostable

Music:

Funkwhale (Code) - federated, self-hostable
Airsonic (Code) - self-hostable

Multi-Factor Authenticator:

Ente Auth (Code) - open source
Proton Authenticator - privacy focused

HTTP(S):

IPFS (Code) - decentralized, self-hostable

Source Code:

Radicle (Code) - decentralized, self-hostable
- User Guide

Mobile OS:

Operating System:

CachyOS (Arch Linux)
Fedora Linux
- Nobara (optimized for gaming / streaming)
Mint (Ubuntu Linux)

Export All Your Google Data

...using https://takeout.google.com/ (optional for Google Photos: GooglePhotosTakeoutHelper) and delete your account.

Converting Google Docs files to Markdown

Install Pandoc to convert docx to markdown
Change dir to your Google Drive export: Takeout/Drive
Create a lua script named fix_underline_links.lua with this content:

function Link(el)
  local content = el.content
  if #content == 1 and content[1].t == "Underline" then
    content = content[1].content
  end
  return pandoc.Link(content, el.target)
end

Run this bash command to convert:

$ find . -name "*.docx" -type f -exec sh -c 'pandoc "$0" -o "${0%.docx}.md" --extract-media=./images/ --lua-filter=fix_underline_links.lua' {} \;

Remove all docx files: $ find . -type f -name "*.docx" -delete

Export Your Twitter (X) Data

TODO: Replace Instagram

How to remove your data from Zuckerberg and backup it up to host it privacy focused.

Glossary

End-to-end encrypted means your data is en- & decrypted only on your machine, resulting in maximum security with maximum responsibility (password loss equals data loss).

Self-hostable means anyone can run the software by themselves and thus maintain control of their data independently of any central authority.

Decentralized means the software does not need to communicate over the internet with a central server in order to work, instead it is capable of communicating independently via peer-to-peer based networking (very useful when dealing with limited internet availability or censoring governments).

Federated means instances of the software build a so called "fediverse" allowing users to communicate across instances.

More Resources

#tutorial #privacy

A Nuclear Powered Nuclear Waste Dispenser

Sean Pedersen — Wed, 13 Nov 2019 00:00:00 GMT

Nuclear power has many advantages but the few disadvantages are severe. One of the major pain points is leftover nuclear waste produced by nuclear fission power plants. It is highly toxic and stays radioactive for thousands of years, leaving a pandora box packed with nuclear death for future humans to enjoy. To sum it up: We urgently need to clean up after our nuclear shit trail.

The current approach is archaic: We dig a hole somewhere and hope it will not come back to haunt us any time soon. This is naive, wishful thinking. Give it enough time and it will leak into our living environments. So how can we do better? My answer is quite simple and radical: Shoot the nuclear waste straight into space, either into deep space leaving our solar system or directly into the sun. This would remove the problem from mother earth for good.

How? Current rocket technology is not gonna do it (launch costs are prohibitively expensive but we are slowly improving, so that could change in the future allowing for rocket based nuclear waste disposal). Right now we need to come up with something different: Design a thick bullet containing the nuclear waste. Build a giant cannon that is powered by nuclear power. Each launch needs to accelerate the bullet to earth's escape velocity and off goes our thick nuclear waste bullet into deep space.

How to build it? Dig a cylindrical hole housing the cannon down into earth (preferably into some high mountain). Install a coil cannon or rail gun powered by nuclear energy capable of accelerating the waste bullet to earth's escape velocity (~12km/s). Before all that, sit down and calculate where to dig the hole according to the needed trajectory. And there it is a sweet nuclear powered nuclear waste dispenser.

Biggest potential hurdle: This nuclear waste dispenser design only works out if it can dispense more waste than is produced needed to operate it. Figuring that out is key to success and is left as an exercise for the curious reader.

Biggest potential failure: The nuclear waste bullet does not reach escape velocity and comes right back at spaceship earth. Mitigation: Simulate several failures without real nuclear waste but the same bullet case design and a realistic non-toxic replacement material for the waste.

I am really curious if this idea is practical. It would be amazing.

References

Understanding understanding

Sean Pedersen — Mon, 08 Jul 2019 00:00:00 GMT

What does it mean to understand something?

If an agent produces a solution for a problem, does it necessarily follow that the agent understands it? Many people would clearly say no. A chess computer excels at the problem of playing chess but has no understanding of what he does, it’s just an algorithm crunching numbers after all, right?

To really understand a problem from a human perspective means not just being able to solve it but having the ability to explain the HOW in depth to others.

Ok well what does it mean to explain something then? I successfully explained something if another agent understands my explanation. Meaning he is able to solve the explained problem as well (applying the transferred knowledge) and able to explain it successfully to other agents (circular explanation).

Can I explain without language? Can I explain without any prior knowledge or beliefs about the knowledge the agent I am explaining to possesses?

Meta Design: Design of a Design

Sean Pedersen — Sat, 20 Apr 2019 00:00:00 GMT

Why even write a design? The design process animates thinking through ideas deeply before getting lost in the details building them. It provides the chance for different stakeholders to develop a shared vision.

What makes a successful design?
It alone must simply suffice to realize the idea it contains.

Some things to try:

First: start with pen & paper and scribble away, to develop a rough outline
Write concisely and precisely but not too short
Prefer diagrams, sketches and tables over long prose
Use clear & vivid explanations; avoid very long continuous text passages

Top Down Approach

Identify distinct, preferably large, independent modules that solve a coherent problem complex which ideally abstract away highly moving parts (things that may likely change / are unknown). Next outline interactions of the modules to map out the interaction dynamics of the system.

“We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each module is then designed to hide such a decision from the others.” - http://sunnyday.mit.edu/16.355/parnas-criteria.html

Finally describe each module as deeply as necessary but not deeper, so that interfaces (behavior towards others) and their functionalities (inner behavior: e.g. algorithms and data structures) are clearly presented. Minimize the information shared with other modules to reduce static coupling (information hiding). Finally draw a clear line between what is solved, what remains unsolved and what is unknowable.

References

https://learnhowtolearn.org/how-to-build-extremely-quickly/

Natural Language Processing: Sentiment Analysis

Sean Pedersen — Sat, 13 Oct 2018 00:00:00 GMT

In this article we will take a closer look at sentiment analysis. Given an input text predict the sentiment (emotional state) as either negative (0) or positive (1). Sentiment analysis is useful to understand human text interactions at big scales. (Updated: 13th January, 2021)

Dataset: IMDB Movie Reviews

A widely used dataset for sentiment classification is provided by the Internet Movie DataBase (IMDB). The training data set consists of 50,000 movie review texts, each associated with the target prediction label 0 or 1.

Models

We compare four increasingly complex approaches to tackle the IMDB sentiment classification problem.

Logistic Regression with TF-IDF & Bi-Grams

N-Grams: Tuples of order N created from a sequence of symbols (words). Bi-gram(a,b,c) = {(a,b),(b,c)}
Term Frequency (TF): Number of occurrences for each word in a document (here a review is a document) of a collection of documents. Results for each document in a sparse (since most likely not all documents share the same words) row vector, where each dimension represents a unique word of the total collection of documents.
N: Number of total documents (reviews).
N(w): Number of documents in which a word w appears at least once.
Inverse Document Frequency (IDF): log(N / N(w))
TF-IDF: TF * IDF

TF-IDF preprocessing results in weighing words heavily that are rare across all documents. The opposite is true for words that are common across all documents. Thus TF-IDF can be interpreted as a heuristic that minimizes the significance of common stop words (the, you, etc.) without needing to explicitly provide a static list of stop words.

Neural Embeddings

Embeddings represent relationships between entities in usually high-dimensional geometric spaces (f.e. euclidean or hyperbolic). One common and useful embedding space for words as entities encodes the semantic / contextual relationships between them. For example one would want words with similar meaning (e.g. synonyms) to be reflected in the embedding by a small distance. A common model to compute embeddings for natural language texts is Word2Vec. Word2Vec can be trained in two ways: either by predicting surrounding words (context) for a given word (Skip-gram) or predicting the missing word from a given context (CBOW).

Instead of training text embeddings unsupervised (without labels) using f.e. Word2Vec: We can train an embedding for our problem at hand as an input layer.
The input layer embedding will learn a problem specific representation (determined by the training data labels and loss function).
The embedding layer works by converting each word-token of an input sequence into a one-hot vector resulting in a big sparse matrix as input, this matrix is matrix-multiplied with an embedding matrix resulting in a dense often low dimensional embedding layer output. The embedding matrix is trained by back-propagated gradient steps from the last layers output computed loss function.

LSTM

LSTM stands for “Long Short-Term Memory” and is a neural architecture designed to model sequential data. It is engineered to overcome the vanishing / exploding gradient problem encountered with classical Recurrent Neural Networks (RNN), enabling LSTM's to model long-term relationships between data points in long sequences. They tackle the fundamental problem of deciding between noise and signal by selectively “remembering” relevant data and “forgetting” irrelevant data with respect to the loss function.

Learn how LSTM's function: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Transformer

Transformers are all the rage for text and even vision tasks now a days. They perform better at modeling sequential interdependencies of their inputs using so called attention heads. Their input size is fixed so they are not recurrent neural networks. Architecturally they are closer related to convolutional neural networks which they are making obsolete for most vision tasks.

Learn how Transformers work:

Deep Learning: Variational Auto-Encoders

Sean Pedersen — Sun, 30 Sep 2018 00:00:00 GMT

Variational Auto-Encoders (VAE) are a probabilistic (variational) extension of classical auto-encoders. Disclaimer: This article does not approach the concept of VAE from a bayesian perspective (check out the references below for this). The focus lies instead on highlighting the key components and their effects on the practical results such as the latent embedding space structure.

Building Blocks of an Auto-Encoder

x: Input tensor (high-dimensional) f.e. image from MNIST dataset (handwritten digits).
Encoder $e (x, θ_{e}) \to z$ : Neural network $e$ compressing input $x$ into a lower dimensional tensor $z$ using parameters $θ_{e}$ .
Latent space z: Hidden lower $n$ -dimensional space (also called bottleneck) in which the compressed representation of input data lies, represented by $z \in R^{n}$ .
Decoder $d (z, θ_{d}) \to x^{'}$ : Neural network $d$ reconstructing a lossy representation $x^{'}$ of the original input $x$ using parameters $θ_{d}$ from a point $z$ generated by $e (x)$ .
Auto-Encoder $d (e (x, θ_{e}), θ_{d}) \to x^{'}$ : Encoder and decoder chained together as a single model.
Loss Function $L (x, x^{'})$ : Reconstruction loss defined by e.g. mean squared error between $x$ and $x^{'}$ : $L (x, x^{'}) = MSE (x, x^{'})$ , and possibly regularization terms e.g. sparsity constraint on $z$ or weights. Used to compute gradients for parameters $θ_{d}$ and $θ_{e}$ minimizing the loss.

Making Auto-Encoders Variational

We turn a classical auto-encoder into a variational one by modifying the encoder. Instead of transforming the input $x$ directly into the single vector $z \in R^{n}$ , we instead map $x$ into two vectors: $z_{μ} \in R^{n}$ and $z_{σ^{2}} \in R^{n}$ . These two vectors parametrize a Gaussian normal distribution (via mean and variance) from which we sample the latent vector $z$ :

z \sim N (z_{μ}, z_{σ^{2}})

This makes our encoder variational (probabilistic), basically adding gaussian noise to our encoder models output vector $z$ .

Why should we add noise to the encoder? Doing so will generate many more different sample points of $z$ for the decoder to learn reconstructions from, forcing the decoder to generate smooth interpolations between local samples in the latent space.

Computing the derivatives of the random gaussian distribution parametrized by the two output vectors $z_{μ}$ and $z_{σ^{2}}$ of the encoder is achieved by reparameterizing into:

z = z_{μ} + \sqrt{z_{σ^{2}}} \cdot ε, ε \sim N (0, 1)

This so called reparameterization trick enables us to take the derivatives of $z$ with respect to either $z_{μ}$ or $z_{σ^{2}}$ , which are necessary to back-propagate the error-signal through the variational sampling layer when using gradient descent as the parameter optimizer.

To prevent the variational encoder of "cheating" by placing different samples far apart from each other in $z$ (avoiding our desired property of smooth local interpolations) we add an additional loss term to our reconstruction loss function, giving the total loss:

L (x, x^{'}) = MSE (x, x^{'}) + D_{K L} (N (z_{μ}, z_{σ^{2}}) ∥ N (0, 1))

This additional loss term is defined as the Kullback-Leibler-divergence (non-symmetric measure of the difference between two probability distributions) between the encoders output $N (z_{μ}, z_{σ^{2}}) \in R^{n}$ and an isotropic standard normal distribution $N (0, 1) \in R^{n}$ . This forces the latent space to be standard Gaussian distributed (achieving the desired smooth local interpolation).

Beta-VAE

By adding a tunable parameter $β$ to the Kullback-Leibler loss:

L (x, x^{'}) = MSE (x, x^{'}) + β \cdot D_{K L} (N (z_{μ}, z_{σ^{2}}) ∥ N (0, 1))

which is $β = 1$ for the normal VAE, we can vary how much we force the latent space to be standard normally distributed (no interdimensional correlations / spherical covariance matrix). This has an interesting effect on the structure of the latent space dimensions: a higher $β$ (>1) promotes disentanglement of the individual latent space dimensions. Making them independent and often interpretable features.
For example a disentangled dimension might represent the abstract feature of gender (negative number may represent male attributes and positive numbers female attributes).

The Underlying Mechanism: Increasing $β$ disentangles the latent features and works by creating an information capacity bottleneck — it makes using any latent dimension "expensive" by penalizing large deviations from $N (0, 1)$ . This constraint forces the model to use each dimension efficiently, making specialization (pure features per dimension) more cost-effective than redundant or mixed encodings across dimensions (penalizing inter-dimensional correlations).

Interestingly, this reveals a broader principle: any constraint that limits effective information capacity per dimension can promote disentanglement. For example, quantizing latent representations creates a similar information bottleneck through precision limits rather than loss induced via $D_{K L}$ , potentially achieving comparable disentanglement effects through the same underlying mechanism of forced dimensional efficiency.

Show me the code

Get your hands dirty and play around:

Beta-VAE

Further Questions

How can we exploit the fact that the decoder is the inverse of the encoder function and vice versa (weights should be in inverse relationship)?

See "Invertible Autoencoders"; TODO: Lookup reversible layers

How is the compressed latent embedding $z$ related with the data distribution of $X$ (f.e. training images)?

„In fact, this simple autoencoder often ends up learning a low-dimensional representation very similar to PCAs." -
http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/

If a simple dense AE approximates PCA, what does a convolutional AE approximate? Local PCAs for each kernel?

Can we train the encoder unsupervised (independently from the decoder) by understanding how the input features are compressed into $z$ ?

The latent space $z$ is shaped by the loss function, which consists of the MSE between input and output image plus the $D_{K L}$ between an isotropic standard gaussian and $z$ . → How can we replace the MSE term of the loss to decouple the decoder from the encoder while training? Define a loss that considers class density and overlap → reward high class density embeddings, penalize multi-class overlap?

References

Excellent detailed blog post: https://lilianweng.github.io/posts/2018-08-12-vae/
Bayesian perspective: https://jaan.io/what-is-variational-autoencoder-vae-tutorial/
Applications: https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf
Intro from the OG authors: https://arxiv.org/pdf/1906.02691