The dragon has slain me - retiring rpglogger.com

skyrimbigi111110080356.jpg

In the two years since Skyrim was released (Skyrim was the original motivation for creating rpglogger.com) I've stopped playing all video games on all platforms. I can't remember the last time bought a new game for my 360, or a mobile game on my phone. As such I'm not really a great person to run, host, or continue development on a project like rpglogger.

So, somewhere around January 1st, 2014 rpglogger.com will be shutting down for good.

Thanks to everyone who tried it out!

Release 102: Internal links

You can now link from a table cell containing text to a WorldObject using the syntax:

[[Some object]][123]

Where "Some object" is the text you want the link to have, and "123" is the ID of the WorldObject you want to link to.

When the user clicks the link they will be taken to the Section that contains the specified object, and the page will automatically scroll to that position.

Release 100: Moving rpglogger from Heroku to EC2

Today I completed the move of the web app for rpglogger from Heroku to Amazon EC2. It was about two days work for this indie developer. I still use Heroku's hosted Postgres database-as-a-service offering for now - in some ways it's pretty great.

Here are the basic steps:

  • Make sure the web app is compatible with both the Heroku and the new AWS stack (i.e. you can deploy the identical app in both places)
  • Bring up and automate the build process of a new server on AWS (in my case, Ubuntu 10.04 LTS)
  • Deploy the app to AWS while it's still live, sharing the database between the two services
  • Add new DNS settings for AWS
  • Stand down the Heroku dynos

Thinking of moving one of your apps from provider to provider? Here's my story.

Compatibility with both environments:

Heroku setup:

  • bamboo-mri-1.9.2
  • Rails 3.2.8
  • Heroku Postgres dev database (10k row limit)

EC2 setup: 

  • RVM w/Ruby 1.9.2-p320 (MRI)
  • Rails 3.2.8
  • Same Heroku Postgres dev database

As you can see my software stack is nearly identical. This helped a lot when my first cold deploy to EC2 went successfully and the app was running identically on the new production server.

Why I decided to keep the database where it is

Should I setup a self-provisioned Postgres server in EC2? No, not now. Three main reasons:

  • I'm an indie developer when it comes to rpglogger. Can I keep my Postgres instance patched and secured as well as Heroku? Probably not.
  • Experience. What if the database experiences some corruption? Do I really want to be the one responsible for fixing that? Nope. Open a ticket with Heroku and let them handle it quickly.
  • Cost. Stayng with Heroku hosted solution is free for now - $0/month until I exceed the 10k row limit on the dev plan. After that I can upgrade to the basic plan for only $9/month. No biggie.

Obviously there's no big gain to moving the database at this juncture, introducing a bunch of unknowns, and effectively recuding the robustness and durability of my database - arguably the most critical part of any web app, and one of the most difficult things to recover from should something really go wrong. Like all good SaaS solutions these days you can grow your costs as you grow your use, and there's a free option to get you bootstrapped.

Sidenote on Heroku Postgres security

One, and only one reason I'm a little concerned about keeping my database hosted on Heroku? They expose the server to the internet directly in order to make it accessible by your app. You can literally log into the instance directly with any Postgres client, from any machine.

Am I all that worried? Well yes, this seems to be a bad choice on Heroku's part, and I'm not sure why hosted Postgres doesn't include an interface to setup access lists or (since their infrastructure runs on AWS anyway) a way to restrict connections to servers within AWS only or in some other way close off direct internet access.

Reasons not to worry too much? First, I don't store ANY sensitive data in this app whatsover. I use 3rd party login (Google, Facebook, Twitter) and I don't store any credentials, payment information, or personal details about any user. The data I actually have is video game strategy guide data. Second, there are daily backups that, while I hope I never have to use them, at least make restoring from something catastrophic pretty easy.

What I really hope? Since hosted postgres is a fairly new service, I hope that ACLs and additional security features are coming soon. Honestly I think the greater danger is that, without even knowing your database password, it's possible to access (and destroy) an entire database with all its backups with only a user's Heroku username and password. There's no second factor authentication, so Heroku is basically trusting that its users will follow security best practices when choosing a password. A good, high security password used to be considered enough security. But these days...hm...not so much.

Ultimately, I could move to something like EnterpriseDB, or ask someone I know who is a professional systems admin to help with with setting up and securing my own Postgres server. For now I'm going to see what Heroku comes up with in the next year while keeping my options open.

Moving the app and adding Capistrano support

One of the nicest things about Heroku is how drop-dead simple it is to deploy. No crazy extra steps or gems to install in your app with dependencies to manage. Heroku just takes care of infering what should be in your stack based on what's in your app and configures your deploy target on the fly as you push code via git. Pretty freakin' sweet.

This means that in order to move to AWS I had to find an equivalent, single-command deploy option. I chose Capistrano, the popular and robust Rails deployment tool.

Setting up Capistrano took an afternoon if you count all the time from the `capify` command to getting the tasks setup the way I wanted. Not too bad. What actually took even longer than that was building, testing, customizing, re-testing, and finally settling on a bootstrap and package dependency install script that would allow me to build a production server from a blank Ubuntu 10.04 LTS image. That probably took another day and a half. So, two whole days to replace the heroku deploy toolchain with something that is nearly as simple to use now that it's up and running.

Another side note: creating a staging server

One major upside to moving to AWS (though I could have done this with Heroku as well, I suppose) is the addition of a staging server to my deployment pipeline to help catch bugs before they hit production. In my case my staging server fits nicely inside a micro instance for about $5-6/month. My two days of Capistrano and shell scripting work paid off here because once I had my staging server built it meant that I could build my new production server in a consistent, automated fashion from a blank OS image within 40 minutes. I'd like to shave that down to somewhere in the 5-10 minute range at some point, but it's not a huge priority at this point.

DNS and connecting the database to the AWS server

Once I had my production server up and running, all that was really left was to setup the database.yml file on the new EC2 instance and point it to the Heroku Postgres database that was already connected to the old Heroku web app. This was as simple as logging into Heroku Postgres and copying the connection settings over to my EC2 server.

After that was working, and I was sure that things were good to go in EC2, all that was left was to swap the DNS settings. This only took a few minutes where I replaced the old A records pointing rpglogger.com to Heroku with my new Elastic IP address on AWS. Because I had exactly the same version of the app running on both Heroky and AWS, and because they both point to the same backend database, Heroku was essentially acting as a second production app server as the changes to the DNS records get propagated across the internet. This is basically a zero-downtime migration for me, and that's great!

What's left?

I'm using the dev plan on Heroku Postgres (10k row limit). At some point in the not too distant future I'm going to hit that limit and need to migration to the basic plan. I'll cross that bridge when I come to it.

rpglogger moving to AWS

This is a minor thing for users of the site, really, but I'm working on migrating rpglogger.com from Heroku's free tier (which actually runs on AWS underneath) onto a reserved instance I've purchased on Amazon Web Services. There are a few benefits that, after the migration, might be interesting for the technically minded:

  • No spin down (and therfore no spin up) delays. The rpglogger app will be up and running at all times, unlike Heroku's free tier which spins down your app after periods of inactivity. This isn't a dig on Heroku - it's actually pretty brilliant the way they've designed their platform - this is simply a fringe benefit.
  • Quicker overall response times for two reasons. First, because I'll be going from one webserver instance (in Heroku parlance they're called "dynos") to two webserver instances. It's not that rpglogger is getting enough traffic to justify two web instances, it's that soon I'll be utilizing image uploading and that will occupy one of the two instances, keeping another instance free for people who are simply reading content on the site. Second, it's been my experience that the "out of the box" response times on AWS virtual servers (even the micro instances) is faster than Heroku, even after the app is spun up and "warmed up" (with some cache, etc.). It's not that difficult for a simple app to give 100ms response times on AWS if most of what you're returning to the browser is HTML and text records from the DB. Of course, rpglogger does a lot more than simple text, and some DB joins that are currently in place are going to slow this down a bit, but I still expect some performance increases simply by moving the application to AWS.
  • I've had an AWS instnace live for some time running plebd.com, another pet project of mine. Sharing that server with rpglogger has some cost and time saving elements to it that I'm looking forward to as well.

That's about it. I still love heroku, but it's been my experience that once you start to do things like supporting file uploads that the ongoing costs start to increase rather rapidly. It's not that these add-ons aren't worth it, it's that adding complexity to the app involves some technical debt, and I'd rather pay that down using AWS than going deeper with Heroku. It's a personal choice, and one which must be made on a per-application basis, but at this juncture I think it makes sense for rpglogger.

New layout preview

Here's a preview of the new layout currently being worked on for rpglogger.com. It's got several navigation, usability, and readability enhancements, including the beginnings of a responsive design that should work well on mobile devices.

Big thanks to David Antaramian for doing the bulk of the update work, and running with things!

Some of the other visual highlights you'll notice are: 

  • Revamped table layout for data display
  • The logbook navigation has moved to the left side, which is more natural for LTR language readers
  • New top navigation bar

...and a lot of other little tweak that I think add up to a fresh experience on the site. I'm aiming to launch the new layout into production by early September, or sooner.

rpglogger "summer of code" starts tomorrow

Having finished up some critical work on another project, tomorrow I will start my "summer of code" for rpglogger. For the next two months (July and August), all my spare development time will be spent adding some new features to rpglogger, making it easier and more powerful to track your progress in RPGs, as well as building your own strategy guides.

If you're interested in contributing to the project, come on board. I'm definitely open to contributors, pull requests, etc. through the GitHub page.

The three major features to be implemented in the next two months will be:

  • Better data entry: basically I hate having to press "submit" when I add something. The site should just save as I type.
  • Index and search - one unified box into which you can type a search term, and from which you get results fed back to you on the page in real time, without needing to press enter, or wait - it just happens and changes as you type. So, when playing Skyrim, if I searched for "daedra" I should get back every location, item, note, etc. that contained that term, case insensitive, and grouped by type (so all the locations would be together, all the items would be together, and so on).
  • Map support: the ability to upload images of world maps and track the locations you find, such that you would mark places on the map, and when you hover over them they would show the information about that place that you'd recorded. You can already track locations in the game, but they're strictly in table form, and sorted in alphabetical order. So, they're not terribly intuitive to manage.

Actively looking for web and mobile developers

rpglogger.com is a project I plan to build and enhance for many years to come, with tweaks and enhancements delivered a few times a year. I will gladly continue to work on it whether or not anyone joins up.

However, today I got some great news when I realized that we got enough notes on StackOverflow's Open Source advertising post that we're now getting free ad placements for developers from StackOverflow for the next six months!

So if you're a web or mobile developer, take a look at the github repo, and the milestones that we currently have open to see what I already have planned. I'd love to take this experience and put it on tablets and other mobile devices.

I was already planning on spending a month or two this summer enhancing rpglogger, and cranking through several of the open issues - now with additional community exposure, I hope we can further amplify that effort!

Current state of rpglogger.com

Just a quick note. Severl visual updates, and a few tweaks to rpglogger have been done in the last few months as I use it to log my progress through Skyrim.

In my personal log, I've got 427 items across 19 categories (books, journal entries, locations, ingredients, etc.) and it's been a great way to record all that.

There are some things that rpglogger really needs, however:

 

  • Overworld map support - an image that you can add locations to as you discover them
  • The ability to tag items with a location (so you can record where a unique item, for example, was found)
  • Real-time search across your log
  • Sharing/collaborative strategy guides

 

And a couple of other nice things, like supporting login across Twitter, and other login systems. I probably won't get to any of this in the near future, but I plan to resume work sometime in the summer (June or July), and I'm taking pull requests on the GitHub repo.

However long it takes, I do plan on keeping this project up for the next several years, as I'm a fan of the Elder Scrolls games (and RPGs in general). Since Heroku can easily handle this project on their free tier, and I don't think we'll be surpassing the free tier capacity any time soon, I figure rpglogger is a great project to just host and enhance slowly over time.

Early development screenshots for rpglogger

rpglogger.com is a labor of passion by a gamer who happens to work as a web developer.

Here are the first three development screenshots. The site is live, and open users. All you need is a Facebook account to log in. I'm two weeks into development, and there is much functionality yet to come.

First version:

  • Initial navigation and layout
  • Preview of filter/meta boxes
  • Demo, lorem ipsum text

 

 

Second version:

  • Refreshed some of the fonts
  • Added background image
  • Added site shout-out at the top of the page
  • Demo, lorem ipsum text

 

 

Latest version:

  • Refresh on the fonts (now using Google Web Fonts)
  • Re-done layout+navigation
  • Overall look & feel is fairly close to final, mostly working on site functionality from here forward, though there will be a few visual enhancements
  • Actual, live data shown

The reason I created rpglogger

I don't like strategy guides for games like Skyrim.

My favorite thing about games like Skyrim is the discovery and experimentation elements. I started to play on launch day, but then stopped after only a couple hours.

Why would I stop playing one of the best games released in a long time? I realized that I wanted, for the first time, to get into the nitty-gritty of alchemy, crafting, and other things in the game. At first I tried to make a handwritten log of people, places, and ingredients, but that broke down very quickly. The game is so huge, that it's just not a practical way to approach the problem.

So, I spent the last week building rpglogger.com - it's basically a way to log your progress in a sort of "build your own strategy guide" sense. You create a 'log book' where you can add/edit/delete 'sections' in the book (e.g. NPCs, quests, items, locations, whatever you want), and define your own integer/text/string/boolean attributes on those sections. For example, if you want to track ingredients, you can add a 'string' field for the name of the ingredient, and 4 string fields, one for each of the effects it has in alchemy. That's just one example.

So, why didn't I just buy the strategy guide, or use GameFAQs.com? I don't like those approaches for games like this, because I feel it ruins the discovery elements by giving you all the answers. I needed a middle ground between handwritten notes and having all the answers.

So, there it is. One gaming geek's story. Right now Facebook login is supported. I plan on adding Twitter, and OpenID providers pretty soon. Right now you can add as many people/places/things as you want. Soon I'll be adding a realtime text search/filter box to quickly find objects, in addition to tagging support. I also built it specifically with the idea that I'd be running it in full-screen mode on my laptop, in the browser, while playing. So, there's dark background/visual theme, and limited need for scrolling.