Three Days, Two Models, and a Thousand Questions: My FastAI ML Journey

What I Learned While Building Image Classifiers From Scratch

---

Reading about four words into this post will automatically tell you that this post was written by an AI. Yes, it is annoying. Yes I chose convenience over quality. Yes, I feel bad. I threw in a few sentences out of my very own head in italics so please don’t judge me too much.

Over the course of three intense days, I dove headfirst into the world of machine learning with FastAI. My goal? To build working image classifiers, learn by doing, and avoid passively watching tutorials. What followed was a whirlwind of wins, roadblocks, and a lot of late-night debugging. Here’s what happened, and what I learned.

---

Day 1: Chasing Birds Instead of Dogs

I began with the FastAI tutorial, which focuses on creating a dog breed classifier using the Oxford-IIIT Pets dataset. But on the first page, I stumbled upon a link to a Kaggle notebook that proposed a simpler challenge: building a bird vs. not-bird classifier.

Intrigued, I decided to deviate from the plan.

The Kaggle notebook used the DuckDuckGo API to fetch images, but I quickly discovered that the API no longer worked. After some research, I switched to the Bing Image Search API. This required setting up Azure credentials and writing a script to collect bird and non-bird images.

However, once I had the images, I ran into another problem: they weren’t formatted or stored correctly. FastAI expects a certain folder structure, so I used Bash to create directories like /bird and /notbird and moved images accordingly.

Using ResNet-34 for transfer learning, I trained my model across 4 epochs. The results were promising. I even added a neat feature: the ability to input an external image and display it with its predicted label and confidence score overlaid on top.

Cutting edge technology

---

Day 2: Dog Breeds, Batch Sizes, and Bottlenecks

It was time to revisit the original tutorial: the dog breed classifier. But I wanted to do it from scratch. No pre-written notebooks. Just raw code and the Oxford dataset.

The original plan used ResNet-34, but training was painfully slow, even on Colab’s free GPU. I tried switching to ResNet-18 for speed, which helped, but my accuracy plateaued at around 93%.

To optimize:

I reduced batch size to avoid frequent CUDA memory errors. (Look at images only a few at once please)

I tried EfficientNet-B3, which promised better accuracy at similar speeds. (Stop looking. I’ll find someone else to look instead)

I increased image resolution to 300×300. (The images are now clearer)

I added data augmentation with aug_transforms(). (The images are now various different versions of the same image. “Hey look a blur! is that still the same image??”)

I trained for 5 epochs instead of 4. (Practice makes perfect?)

Despite these tweaks, my validation loss remained stubbornly high. Some class labels in the dataset (like Great Pyrenees) seemed mislabeled, further complicating training.

Key lesson: model choice matters, but data quality and compute constraints play an equally large role.

---

Day 3: Frustrations, Frontends, and Future Plans

With two models under my belt, I wanted to push further.

First, I attempted to build a frontend for the bird-not-bird classifier using Replit (btw, if you want frontend, use Lovable. If you want logic, use cursor. Replit is like an ugly middle child that sucks both ways). But loading the trained .pkl model into the environment proved difficult. Replit didn’t play well with FastAI’s export structure, and the frontend never fully came together.

Then, I explored a new idea: a car brand classifier using a Stanford dataset. While I managed to begin data prep and explore the set, I ran out of time before completing model training.

Still, the seeds are planted.

---

Lessons Learned

Start with code, not videos. FastAI is best learned by doing.

APIs break. Be ready to adapt.

Folder structure matters more than you think.

ResNet-18 is fast, but has a ceiling. Go bigger if you can.

Don’t underestimate data quality. Labels can sabotage your results.

Colab’s GPU is your friend, but memory constraints will require workarounds (like batch size tuning).

Use vision_learner, DataBlock, and aug_transforms() wisely. They give you flexibility without requiring a PhD in ML.

---

Next Steps

Looking ahead, I’d like to:

Finish the car brand classifier.

Explore a movie recommendation system.

Deploy a working frontend.

Help at least one other learner build their own model.

If you’re just starting out: don’t wait. Open a notebook, ask questions, and start building. You’ll learn more from one failed run than ten perfect videos.

---

Got questions or stuck somewhere? I’m happy to help. Reach out or drop your notebook link.