Building WoWdb

9 months agoReact Native, Search, World of Warcraft

Building WoWdb
WoWdb is a mobile World of Warcraft database. It's built with React, React Native, Fastify, MongoDB, and hosted on Render.

I built Mr. Bigglesworth, an app that would let you search for World of Warcraft content and view Wowhead comments for them. It was based on an in game cat.

It was a pretty simple solution. I wrote a script in Node.js, which would use the Blizzard APIs to fetch data. Which I then transferred to my MongoDB Atlas database. I also wrote an API in Node.js, which would search through the items, quests, achievements, etc, and return to the mobile app, along with comments from Wowhead. The mobile app was built with React Native.

It was a decent setup but had its drawbacks.

Problems

In 2019, I plan to retire some of my old apps that have no installs or don't make sense anymore. And focus more on the ones that do.

As part of this restoration effort, I decided to rebuild Bigglesworth with a few changes.

Changes

Finding the right infrastructure

A major hurdle to the ideal setup was my infrastructure.

I was hosting the API on Heroku, which would connect to MongoDB and fetch data, search, and hit up Wowhead for comments. It was fast and it worked. But that was it. It was expensive, so I was still doing the data updates manually, and rarely.

Recently, Google Cloud introduced Cloud Scheduler, and it was cheap. I looked into it. But before I could finalize it, I came across Render. Suddenly, so many new doors opened.

To improve search, I wanted to use Algolia. But my project didn't fit their criteria for open-source and had enough data to cost $500 a month. Which, WoWdb being a side project, I just didn't wanna spend. So I decided to write my own little service just for search.

The ideal setup

I sat down and started to draw out what the ideal setup would be like. My requirements were simple.

Luckily, Render offered everything in one place. Cronjobs, microservices, static site hosting. And cheap, too!

Now that I could build everything in one place, I got to work.

Rebuilding

First, I rewrote the data miner. Blizzard recently sunsetted their old APIs. So I had to fix my script to fetch from the new API. Improvements were also made. Decided to add spells, a missing collection of data, to the miner.

Then I wrote the new search microservice. It loads all data from MongoDB and uses FuzzySearch to run queries. I also tried the more popular Fuse, but couldn't get decent performance. Not sure if it was my configuration or just the sheer size of my dataset. Roughly 550k records! Average search time with Fuse was 1 second. It was a mere 100 milliseconds with Fuzzy!

When this effort started and before I had decided to run everything on Render, I updated the Heroku app with a v2 API. But soon realized it was a mistake because things were changing considerably and it would simply be cleaner to leave the Heroku app as-is and drop all v1 code and deploy v2 straight to Render.

The new API removed search from it. It just passed the query to the microservice and returned the data to the client. Refactored a ton of code. Just general improvements and learnings since last year when I first wrote it.

The mobile app was probably the largest change. New views, bug fixes, design, etc. Building the detail views for items, mounts, achievements, etc, was the trickiest part. A lot of time and testing went into that. It was pretty much a complete rewrite. Which I always love!

I'm quite pleased with the results. It's a very neat setup. Everything is updated to the latest version of everything. It was incredibly fun to work on. But it's still not perfect.


There are still a few improvements that can be made.

Head on over and check out the updated app. As always, my side projects are completely open-source. You can find the code on GitHub.