🧠 NeuralNetworkBasics

Building neural networks from first principles — no magic, no black boxes.

A hands-on implementation of computation graphs, manual derivative calculations, and backpropagation using pure calculus and the chain rule.

---

✨ What This Project Is About

Most deep learning tutorials hand you PyTorch or TensorFlow and say "trust the .backward() call."

This project does the opposite.

Every gradient is computed by hand, every node in the computation graph is explicitly differentiated, and every chain rule application is written out in code. The goal is to build genuine intuition for how automatic differentiation actually works under the hood — before letting any framework do it for you.

"Everything became much clearer when I started ignoring full-page dense derivations and just started writing code." — Andrej Karpathy

---

🌿 Branch: feature/manual_derivative_calculation_for_each_node

This branch is the mathematical core of the project. It implements:

• A Value class that wraps scalars and tracks operations to form a computation graph
• Manual derivative formulas for each operation node using calculus rules
• Local gradients for direct parent–child relationships
• Chain rule propagation to carry gradients backwards through indirect dependencies
• Numerical gradient verification — comparing analytical derivatives against finite differences to catch errors
• The foundation for automatic differentiation (autograd) built from scratch

---

🗺️ Concepts Covered

1. The Value Class & Computation Graph

Each scalar in the network is a Value node. When you perform operations on Value objects, they silently record the computation graph — who produced whom, and which operation was used.

a = Value(2.0, label='a')
b = Value(-3.0, label='b')
c = Value(10.0, label='c')
e = a*b; e.label = 'e'
d = e + c; d.label = 'd'
f = Value(-2.0, label='f')
L = d * f; L.label = 'L'

This builds a directed acyclic graph (DAG) of operations — exactly what modern autograd engines use internally.

---

2. Manual Derivative Calculation

For each node, the local gradient (how much the node's output changes with respect to each of its inputs) is derived analytically using calculus rules:

Operation	Local Gradient
z = a + b	∂z/∂a = 1, ∂z/∂b = 1
z = a * b	∂z/∂a = b, ∂z/∂b = a
z = a ** n	∂z/∂a = n · aⁿ⁻¹
z = tanh(a)	∂z/∂a = 1 − tanh²(a)
z = exp(a)	∂z/∂a = exp(a)
z = a / b	∂z/∂a = 1/b, ∂z/∂b = −a/b²

No approximations. No finite differences. Pure calculus.

---

3. Chain Rule Propagation

The chain rule is what makes backpropagation possible. For any intermediate node e in the graph:

dL/da = (dL/de) · (de/da)

In code, this means every node multiplies the gradient flowing in from its children by its own local gradient, then passes the result upstream to its parents.

This accumulation (using +=) is critical — a single node can feed multiple downstream paths, so gradients from all paths must be summed.

---

4. Numerical Gradient Verification

To validate that our analytical derivatives are correct, we compare them against numerical derivatives computed via finite differences:

df/dx ≈ (f(x + h) - f(x)) / h     where h → 0

If the analytical and numerical gradients match (within floating point tolerance), the implementation is correct. This is a standard technique used in industry to unit-test gradient implementations.

---

🏗️ Project Structure

NeuralNetworkBasics/
│
├── init.ipynb    # All concepts: Value class, computation graph, manual backprop & gradient verification
└── README.md

Everything lives in a single self-contained notebook — no scattered scripts, no dependencies to wire up. Open it and run top to bottom.

---

🚀 Getting Started

Prerequisites

pip install numpy matplotlib jupyter

Clone & Run

git clone https://github.com/FazalHussain/NeuralNetworkBasics.git
cd NeuralNetworkBasics
git checkout feature/manual_derivative_calculation_for_each_node

jupyter notebook init.ipynb

---

📐 The Math Behind It

Chain Rule (Multi-Variable)

For a node Z that depends on y, which depends on x:

Derivative

Derivative Calculation

📖 References: Chain Rule — Wikipedia · Derivative — Wikipedia

---

🎓 Learning Path

This project follows the philosophy of learning by building. The recommended progression:

1. Understand scalar derivatives         
2. Build the Value class                 
3. Construct a small expression graph   
4. Derive local gradients manually      
5. Apply chain rule backwards
6. Verify with numerical gradients      
7. Generalize → autograd engine         →  (next milestone)

---

🙏 Acknowledgements

This project is deeply inspired by Andrej Karpathy's legendary lecture series on building neural networks from scratch:

📺 The spelled-out intro to neural networks and backpropagation: building micrograd

Karpathy's micrograd project demonstrates that a complete autograd engine — capable of training real neural networks — can be implemented in under 100 lines of Python. This repository walks through the same ideas with additional annotation, manual derivation steps, and verification tooling.

---

🗺️ Roadmap

• Value class with computation graph tracking
• Manual derivative calculation for each operation node
• Chain rule propagation (manual backpropagation)
• Numerical gradient verification
• Generalized backward() pass (topological sort)
• Neuron, Layer, and MLP classes built on top of Value
• Training a small classifier end-to-end
• Visualization of computation graphs

---

👨‍💻 Author

Syed Fazal Hussain Naqvi Senior Android Engineer & Technical Lead · AI Engineering Enthusiast

---

"To understand backpropagation, you must first understand differentiation. To understand differentiation, you must write it out yourself."

⭐ Star this repo if it helped you understand the math behind neural networks.