Building WoWdb

I built Mr. Bigglesworth, an app that would let you search for World of Warcraft content and view Wowhead comments for them. It was based on an in-game cat.

It was a pretty simple solution. I wrote a script in Node.js, which would use the Blizzard APIs to fetch data. Which I then transferred to my MongoDB Atlas database. I also wrote an API in Node.js, which would search through the items, quests, achievements, etc., and return to the mobile app, along with comments from Wowhead. The mobile app was built with React Native.

It was a decent setup but had its drawbacks.

Problems

• Search was a poor experience; MongoDB text search is rather limited and has no typo tolerance or scoring algorithm. So any searches for guldan would return nothing because the data stored is Gul'dan.
• App Store and Google Play reviews were bad. Their main grief was that it only lets you search and view Wowhead comments and not see the objects' details. Which, to be fair, was the intended and marketed purpose of the app.
• Because the data fetching and updating was a manual process, I hadn't updated it since I first released the app over a year ago. WoW releases new content all the time, and some data just wasn't there to be searched.

In 2019, I planned to retire some of my old apps with no installs or don't make sense anymore. And focus more on the ones that do.

As part of this restoration effort, I decided to rebuild Bigglesworth with a few changes.

Changes

• Let users see details about items, mounts, quests, etc. As much as I can get from the Blizzard APIs.
• Better, faster search with typo tolerance.
• Because I couldn't find someone who could do a Mr. Bigglesworth icon for me cheaply, I decided to rebrand the app to WoWdb. I chose that capitalization as not to conflict with WoWDB, a competitor to Wowhead.

Finding the right infrastructure

A major hurdle to the ideal setup was my infrastructure.

I was hosting the API on Heroku, which would connect to MongoDB and fetch data, search, and hit up Wowhead for comments. It was fast, and it worked. But that was it. It was expensive, so I was still doing the data updates manually and rarely.

Recently, Google Cloud introduced Cloud Scheduler, and it was cheap. I looked into it. But before I could finalize it, I came across Render. Suddenly, so many new doors opened.

To improve search, I wanted to use Algolia. But my project didn't fit their criteria for open-source and had enough data to cost $500 a month. Which, WoWdb being a side project, I didn't want to spend. So I decided to write my own little service just for search.

The ideal setup

I sat down and started to draw out what the ideal setup would be like. My requirements were simple.

• A place to host the REST API
• A microservice for search
• A weekly cronjob for updating data
• Hosting for a single page React landing page

Luckily, Render offered everything in one place. Cronjobs, microservices, static site hosting. And cheap, too!

• $5 a month for web and microservice each
• $2 a month for cronjobs
• Free static sites

Now that I could build everything in one place, I got to work.

Rebuilding

First, I rewrote the data miner. Blizzard recently sunsetted their old APIs. So I had to fix my script to fetch from the new API. I decided to add spells, a missing collection of data, to the miner.

Then I wrote the new search microservice. It loads all data from MongoDB and uses FuzzySearch to run queries. I also tried the more popular Fuse but couldn't get decent performance. Not sure if it was my configuration or just the sheer size of my dataset; roughly 550k records. The average search time with Fuse was 1 second. It was a mere 100 milliseconds with Fuzzy!

When this effort started, and before I had decided to run everything on Render, I updated the Heroku app with a v2 API. But soon realized it was a mistake because things were changing considerably, and it would simply be cleaner to leave the Heroku app as-is and drop all v1 code and deploy v2 straight to Render.

The new API removed search from it. It just passed the query to the microservice and returned the data to the client. I refactored a ton of code—just general improvements and learnings since last year when I first wrote it.

The mobile app was probably the largest change—new views, bug fixes, design, etc. Building the detailed views for items, mounts, achievements, etc., was the trickiest part. A lot of time and testing went into that. It was pretty much a complete rewrite. Which I always love!

I'm quite pleased with the results. It's an elegant setup. Everything is updated to the latest version of everything. It was incredibly fun to work on. But it's still not perfect.

There are still a few improvements that can be made.

• Render currently only hosts services in the US west. My MongoDB Atlas cluster is in Ireland. Most of the app's potential users are also in the EU. The latency is a huge problem right now. Luckily, it's on their roadmap to add support for the EU and other regions. But until then, it's a bit slow for the rest of the world.
• Blizzard APIs have limited data and even fewer attributes than would be beneficial. Wowhead, on the other hand, data mines the game files and has way more data than I can get from Blizzard APIs. Perhaps in the future, I could crawl Wowhead instead. But that's a slippery slope and a conversation for later.
• I lost my Android signing key! Luckily, I had the app enrolled in Google Play Signing, and their support team was accommodating in updating my key. The Android update should be live over the next few days. The iOS update is already available on the App Store.
• I want to redesign the app's visuals to match the Battle for Azeroth website. I think it's fantastic artwork, which people will really appreciate. I redesigned my guild's website last year from the Legion design, and it was a hit.

Head on over and check out the updated app. As always, my side projects are completely open-source. You can find the code on GitHub.