What if your AI could learn from every conversation without losing its ability to think logically?

That’s the problem I set out to solve with the Synaptic Switch—a new neural network architecture that challenges how we think about memory and learning in AI systems.

The Problem Every AI System Faces

Here’s something frustrating about current AI: it can’t truly learn from you without potentially breaking. If you train a language model on new conversations, it might forget how to do math. Teach it new facts, and it might lose its reasoning abilities. This phenomenon is called "catastrophic interference," and it’s been a thorn in the side of AI researchers for decades.

The usual workaround? Don’t let the AI learn at all during conversations. Instead, store everything externally in vector databases or context windows, and hope the model can piece it together. It works, but it’s like giving someone a notebook instead of letting them actually remember things. It’s memory as a crutch, not as a fundamental capability.

A Different Approach: What if Neurons Could Choose?

The core insight behind Synaptic Switch is simple but powerful: what if we separated what the AI knows from what it’s learning right now?

Think about how you remember things. You have stable, long-term knowledge (like how to read, or that 2+2=4) that doesn’t change when you have a conversation. But you also have working memory that adapts in real-time—you remember what we talked about five minutes ago, you pick up on conversational patterns, you adjust to the person you’re talking with.

The Synaptic Switch mimics this with a dual weight system. It has "slow weights" that capture structural knowledge—the fundamental logic, language understanding, and reasoning capabilities. These are trained carefully and remain stable. Then it has "fast weights" that act as a scratchpad for immediate, episodic learning. These update constantly using a biological learning rule called Hebbian plasticity: "neurons that fire together, wire together."

How It Actually Works

Here’s what makes it different from traditional approaches:

First, there’s the dual weight architecture. Slow weights (W_p) handle the heavy lifting of language and logic. They’re trained through standard backpropagation and stay relatively fixed during conversations. Fast weights (W_f), on the other hand, are volatile and adaptive. They capture the immediate conversational context—your speaking style, recent topics, specific preferences—without touching the core knowledge.

Second, everything stays in continuous vector space. Instead of forcing outputs into discrete tokens (like choosing the word "cat"), the network maintains information as high-dimensional continuous vectors. This might sound technical, but it matters: continuous representations preserve nuance that gets lost when you force everything into discrete symbols. It’s the difference between a spectrum and a checkbox.

Third, there’s a clever filtering mechanism called the Plastic Memory Layer (PML). Think of it as a dynamic lens that can be switched on or off. When you’re doing pure reasoning, the PML is bypassed—you get clean, unmodified logic from the slow weights. When you’re in conversation mode, the PML activates and personalizes the output based on what the fast weights have learned. It’s not generating new information; it’s refining the latent representation based on recent experience.

Fourth, and perhaps most importantly, there’s homeostatic decay. This is the network’s built-in cooling system. If the fast weights start accumulating too much energy (imagine someone obsessing over a single topic for hours), the decay rate automatically increases. This prevents hallucination, topic obsession, and the kind of drift you see in models that over-adapt. It handles topic transitions naturally without needing manual resets.

What This Means in Practice

The practical implications are significant. You can train the slow weights once on a large, diverse dataset to establish robust reasoning and language capabilities. Then, during deployment, the network learns from conversations through the fast weights. A user talks to the AI for an hour, and it genuinely adapts—it learns their communication style, remembers context, picks up on preferences. But if you test its math abilities or logical reasoning, they’re unchanged. The structural knowledge remains intact.

When you switch conversations or users, you simply reset the fast weights. It’s like clearing your working memory but keeping your long-term knowledge. The base model remains stable and reliable while still being adaptive where it matters.

Why This Matters

Catastrophic interference has been a fundamental barrier to continual learning in AI. Solutions typically involve external memory systems, careful data replay, or avoiding learning altogether. The Synaptic Switch proposes something different: structural modularity at the weight level. By explicitly separating stable knowledge from adaptive memory, we can have both reliability and flexibility.

This isn’t just theoretical. The implementation combines high-performance C code for the core computations with a clean Python API. You can train a model, have it learn from conversations, verify that its logic is preserved, save and load personalized states, and monitor the homeostatic dynamics in real-time.

The Bigger Picture

The Synaptic Switch is part of a broader exploration I’ve been doing into memory-native architectures—neural networks where memory isn’t bolted on but is intrinsic to how the neurons work. It’s an experimental cousin to the Memory-Native Neural Network (MNNN) family, which explores similar questions through different architectural choices.

Where MNNN focuses on making memory intrinsic to individual neurons through mechanisms like adaptive time constants and associative manifolds, Synaptic Switch tackles the catastrophic interference problem head-on with explicit weight separation and Hebbian learning. They’re different approaches to the same fundamental question: what if memory wasn’t something we added to neural networks, but something that was already there in how we design the neurons themselves?

Try It Yourself

The entire implementation is open source and available on GitHub: github.com/hejhdiss/SYNAPTIC-SWITCH-NEURAL-NETWORK

The repository includes the C library, Python API, comprehensive documentation, and working demonstrations. You can experiment with the dual weight system, watch homeostatic decay in action, and see for yourself how the network preserves logic while learning from conversations.

This is experimental research-grade code. It’s meant to be played with, tested, broken, and improved. If you’re interested in continual learning, memory-native architectures, or just want to experiment with a different way of thinking about AI memory, I’d encourage you to check it out.

What’s Next?

The Synaptic Switch raises as many questions as it answers. How well does this scale to larger models? Can we develop more sophisticated Hebbian learning rules? What happens when we combine this with transformer architectures? Could this approach work for multi-modal learning?

These are open questions, and that’s exactly the point. This isn’t a finished product—it’s an exploration, a testbed for ideas about how neural networks might handle memory and continual learning differently.

The code was generated through conversations with Claude Sonnet 4.5, which itself is fascinating—using AI to explore new architectures for AI. But the ideas, the experiments, the countless iterations to make it work—that’s been a journey of exploration into what’s possible when we challenge our assumptions about how memory should work in neural systems.

If nothing else, I hope this demonstrates that there are still fundamentally different ways to think about core problems in AI. Catastrophic interference isn’t solved by better external memory or more clever training schedules. Maybe it’s solved by rethinking what memory means at the architectural level.

---

---

Built with curiosity, powered by Claude Sonnet 4.5, driven by the question: Can we build AI that learns continuously without forgetting?