Designing a Video Game API

We’ve talked a bit about Spycursion and its open source model, but we haven’t really delved into how that will work from a practical standpoint. In this post I’ll talk about application programming interfaces, or API‘s, with the goal of helping you understand some of the decisions that go into designing an API for a multiplayer game like Spycursion. Please note, this information is intended for a more general audience, so many of the details have been simplified. This post is an introduction, not a reference manual!

What is an API?

An API is a set of rules and agreements, through which developers of various software components agree how those components will communicate. You can search the internet for API documents of all sorts to get a feel for what typically goes into those documents, but if you’ve never tried writing code for an API before (or writing code at all), you may get lost in the jargon.

Instead, let’s look at a simple real-world example…

Server: Hello, and welcome to Foodz R Us! Can I get you anything to drink?
You: Purple cow.
Server: Excuse me?
You: Purple cow.
Server: Err, I’m… sorry, we don’t have that on the menu. Would you like-
You: PURPLE COW!

When a server at a restaurant asks if you’d like anything to drink, they are usually expecting you to name something on the drink menu, or perhaps to ask a question. Here, though, you violated their expectations and the server’s “API” returned an “error.” (I personally have never seen “purple cow” on any drink menu, but if you do, let me know!) In computer-speak, this example could be converted to something like…

Server:

{ "request": "drink_choice" }

You:

{ "drink_choice": "purple_cow" }

Server:

{
  "error": "purple_cow not found",
  "drinks": [ "water", "tea", "milk", "juice", "beer" ]
}

{ "request": "drink_choice" }

You:

{ "drink_choice": "purple_cow" }

Server:

{
  "error": "purple_cow not found",
  "drinks": [ "water", "tea", "milk", "juice", "beer" ]
}

{ "request": "drink_choice" }

You:

{ "drink_choice": "purple_cow" }

Seasoned programmers will recognize that the messages above are in the popular JSON format. Certainly not all API’s use JSON (Spycursion’s does), but I will be using it throughout this post, for its simplicity and readability.

Movement

Just about every multiplayer video game will, at some point, involve you moving one or more characters from point A to point B on a game map. To keep things simple, let’s say our game map is a 2D grid made up of integer coordinates. (You know, the grids your high school Geometry teacher made you draw.) When you first start the game, you don’t necessarily know your character’s starting point, but the server should have that information, and so it lets your game client know:

{ "player_position": [ "x": 5, "y": -2 ] }

Now that the client and server are synced, you can move your character around the map. In a flat, boring world with no obstacles whatsoever, your client could just do something like this:

{ "move": [ "x": 3, "y": 1 ] }

But where’s the fun in that? In Spycursion, as in most other games, pathfinding may be required. Walking directly from A to B in a straight line might have you plowing through buildings or getting run over by a car! So the client, or the server, or both, will need to be able to navigate your character around obstacles. In practice, you would likely rely on the client for pathfinding, then convert that into API requests. If you as a player click the point (3,1), but there is an object in the way, the actual API requests might look like this:

{ "move": [ "x": 6, "y": -2 ] }
{ "move": [ "x": 6, "y": 1 ] }
{ "move": [ "x": 3, "y": 1 ] }

But what happens if a player hacks their client to just ignore those obstacles? What does the server do with a request that seems to defy the laws of physics? This is where collision detection comes into play, and any well-designed game server will include it. Basically, upon a player trying to run through a tree, the server sends a message to said player’s client…

{ "player_position": [ "x": 5, "y": -2 ] }

… as if to say, “Sorry, that thing is impermeable. You’re still at (5,-2).”

REST

No.

… Sorry, I should explain. REST stands for representational state transfer and is very often used in a sentence with “API.” Without getting into too many details, REST is an architectural style that is common among web API’s, and one of its more notable features is that the server does not store any client context between requests. In other words, every client request has all of the information that the server needs in order to process it. In a RESTful API, movement requests would look something like this:

{
  "from": [ "x": 5, "y": -2 ],
  "to": [ "x": 6, "y": -2 ]
}
{
  "from": [ "x": 6, "y": -2 ],
  "to": [ "x": 6, "y": 1 ]
}
{
  "from": [ "x": 6, "y": 1 ],
  "to": [ "x": 3, "y": 1 ]
}

A little verbose, but not terrible, right? Well, think about that little cheater with the hacked game client:

{
  "from": [ "x": 5, "y": -2 ],
  "to": [ "x": 3, "y": 1 ]
}
{
  "from": [ "x": 6, "y": -2 ],
  "to": [ "x": 4, "y": 7 ]
}
{
  "from": [ "x": 666, "y": 666 ],
  "to": [ "x": 666, "y": 666 ]
}

A stateless API would give the kid free reign to teleport around the game map! Naturally, this makes no sense for Spycursion, and you should treat any multiplayer game advertising a “REST API” with extreme skepticism.

Vision

I don’t believe we’ve talked about this, but here’s a new Spycursion tidbit for you — every player and NPC in the game has a vision cone. You can only see characters about 135 degrees in front of you (buildings and other stationary objects are always visible), so it matters which direction you’re facing. If you sneak up behind another player, they can’t see you without turning around. The API request to change where you’re facing is straightforward:

{ "look": 1.570795 }

You might recognize that number — yes, here are those Geometry nightmares again — as half of pi, telling the server that your character is now looking at pi/2 radians, or 180 degrees, which in this case just means “due south.” When you turn around, you might discover another player trying to sneak up on you, about which the server now sends a warning:

{
  "player_id": 66666,
  "player_position": [ "x": 5, "y": -3" ],
  "player_looking": 0
}

In reality, you wouldn’t be sent the opposing player’s actual ID (see below), but instead a lot of information about their appearance, and perhaps an indication that you’d encountered them before. Again, we’re keeping things simple here.

Interaction

Another MMO trope: Interacting with other characters. (I know, it’s awful.) In most games, if you wanted to converse with and/or hit on another character, your client might send a request like this:

{
  "action": "talk",
  "player_id": 66666,
  "message": "Hey, stop sneaking up on me."
}

And that’s all well and good, if your client has the player’s ID, which it probably does, as it was probably already sent by the server. Spycursion, though, is a bit different. We can’t tell you a given player’s actual ID or username, because of the possibility of disguises. (This is a spy game, after all!) If you knew that little Johnny’s player ID was 66666, and this fact was always shared by the server, then little Johnny could never truly fool you. Instead, we have to assign a temporary ID for every character who enters your vision cone. This is shared between client and server, until enough time has passed, or you leave the map, at which point you “forget what they looked like.”

If another player starts a conversation with you, your client gets a message like this:

{
  "player_id": 1234,
  "says": "I heard there was a new Spycursion blog post today!"
}

Other spy-like requests look similar, except of course the server doesn’t notify your client that someone has just planted a bug on you. 😉

Hacking

This post wouldn’t be complete without me talking about the API details behind what happens on your character’s computer screen. That said… there actually isn’t much to talk about! Spycursion’s in-game internet will offer many services which can essentially be divided into two types — “hackable” and “unhackable.” The “unhackable” services will have their own API specifications, while the “hackable” services will consist of in-game programs written in Slang. All of the Slang programs follow one simple API request:

{
  "command": "rm",
  "args": [ "-rf", "/" ],
  "target": "0123:4567:89ab:cdef:0123:4567:89ab:cdef",
  "session_key": "0123456789abcdef"
}

The target and session key are just for verification that you are logged into a remote host, and for allowing you to have multiple terminals running at once. That request includes every (non-client-specific) command you’ll run from the terminal, including network tools, the Slang compiler, and your programs themselves.

We haven’t fully decided which services will fall under which categories. Transparently, we would love every service to fall under the “hackable” category, but it’s just a question of feasibility and what we actually want players mucking around with.

 

Spycursion’s API documents aren’t yet ready for public release (and even if they were, the server isn’t ready for public release, so you wouldn’t be able to do much with them). In the meantime, hopefully this post has taught you a few things about video game API’s, and maybe even motivated you to design your own!



Some things we’ve learned developing a 3D game for Lisp.

Hi, Mauricio Here,

This blog post is kind of a developers blog, as I said in my only previous post I’ve been dealing with the MS Windows side of things, I also made some 3D models using Blender, and use the abstractions Scott had created to create a a new scene for the game (a bedroom), but of course my job has not been limited to that. We’ve learned some stuff that we would like to share, so that is what I am doing.

Classimp on Windows.

One of the earlier lessons we learned was related to Assimp, for one thing downloading the binaries and moving the dll to your project directory does not work, in recent Assimp versions the name of the dll isn’t even one of the names searched by Classimp, and renaming or making the code match the dll name does not do the trick. In order to make classimp work we had to compile with gcc instead of visual c, using mingw,, this produces a libassimp.dll which is what classimp looks for and that DLL works fine. The latest Assimp library supported by Classimp is 3.3.1, so if you want to compile it yourself download that version and then follow the instructions here.

When you compile with MingGW using the instructions in the link the DLL will be in the “assimp-3.3.1\build\tools\assimp_cmd” directory, and asides from that you get an assimp.exe executable that is very useful for asset conversion. Currently we are consuming fbx files, so when I created models in Blender I exported them as fbx files, and that caused problems in some cases, so we adopted a different  workflow which so far guarantees a model that loads correctly in CEPL, we use assimp.exe in the command line to convert the .blend to .dae which is the extension for Collada, an open exchange format, and then use Autodesk FBX Converter to convert the .dae to .fbx. This workflow gives you better control over the way that the files are converted.

Why not reading the .blend files directly?, Well for one thing we had some pre-made assets and those were fbx, and we want to be uniform., another reason is that sometimes the file would not be read correctly by classimp, but assimp.exe tries several stuff and is able to export it, as long as you remove any lamps form the scene (I also remove the cameras for good measure). And a third reason, is that sometimes we want to pre-transform the vertices  specially with several objects in the same file, and doing this during the export simplifies the code to read the file. So we do use this:

assimp export file.blend  file.dae -ptv

In the command line, where -ptv stands for pre-transform vertices, and assimp guesses the output file format from the extension of the second file. Assimp does not export fbx , that is why we export to ,dae and then convert. Classimp can read .dae files correctly, but fbx files are smaller, and the difference in load time is negligible for our purposes, and as I said we want to keep uniformity.

Another good tip is to use the assimp viewer to open an asset, and then look at the log file, this will warn you of possible problems when importing with classimp.

Multiple Textures in one Mesh.

We ran into a problem with a model of an avatar we were using in the game, the textures were not imported correctly, the head texture was applied to the body, so the model looked like it had no pants, and it had some weird things in the skin (actually stretched head features). The problem is that when you run (classimp:import-into-lisp) each mesh object contains only one texture. however the model had a mesh with two textures.

Luckily Blender allows you to select the portion of the mesh that is assigned to a particular texture and separate it. So we separated the head from the body, re exported, re-sorted the textures and voila, problem solved.

How do you know the order of the meshes? Well, the first thing is to name things right in Blender, then the structure you get when you import into lisp has a tree and that gives you the clue.

SPYCURSION-CLIENT> (ai:import-into-lisp (car *char8-data*))
#<CLASSIMP:SCENE {100A0E4AE3}>
SPYCURSION-CLIENT> (ai:root-node *)
#<CLASSIMP:NODE {1004FA9443}>
SPYCURSION-CLIENT> (ai:children *)
#(#<CLASSIMP:NODE {1004FA8663}> #<CLASSIMP:NODE {1004FA8903}>
  #<CLASSIMP:NODE {1004FA8B93}> #<CLASSIMP:NODE {1004FA8E33}>
  #<CLASSIMP:NODE {1004FA90D3}> #<CLASSIMP:NODE {1004FA9373}>)
SPYCURSION-CLIENT> (loop for x across * collect (ai:name x))
("Head" "Gadget" "Bag" "Body" "Jacket" "Hair")
SPYCURSION-CLIENT>

So the first mesh is the head, the second the Gadget, and so on. The pre existing textures worked well because I just separated and saved again so the UV coordinates remain intact,  now each mesh reads from the right image, and maps the right coordinates.

Blender

One of the models I was making was a full room, to do it I used Archimesh, the room was simple, a rectangular room with a window and a door,  and some cabinets, this worked out perfectly in Blender, but not so much when I imported the models in CEPL. The thing is that boolean operators produce weird geometry, and the holes in the walls for the door and window are produced by boolean subtraction. In other models the sub surface modifiers looked like crap when imported,

So I decided to re-make the room the old fashioned way, loop-cuts and face deletion, replace the old room, and apply all of the modifiers to other objects before exporting, that way the look was predictable.

The other thing I learned is that you need to either use the a processing flag in classimp or export the model with pre-transformed vertices, otherwise the objects in the scene were not placed in the right position. I used Blender to lay out the room furniture, and then exported each element separately, that made it easier to place and remove things, and handle the textures.

I also spend some time baking textures, and exporting the UVs, the UVs exported from Blender are immediately usable, and you can use DIRT to import the texture, so asides from the tedious process of UV wrapping the objects, assigning the image, and baking the textures into it, the process is very painless.

Final Thoughts

After ironing out the exporting problems saving models so they can be used with CEPL turns out to be a straight away process, there are some things in our code we are optimizing to make the process of maintaining the software and adding stuff more straight forward, but we have a working skeleton,  we can create rooms and cities, place avatars, animate them,  etc.

Of course there is still work to do, and Spycursion is curious in that it has two environments, the OS and the 3D world, but even when we some times hit ourselves against a wall, there is always a door a couple of steps away, as we learn more is like turning the lights on, we now where the walls doors, windows and other stuff is, and that allows us to move freely.



Creating a (Non-Trivial) Lisp Game in 2018

Lisp programmers are a small group. According to the TIOBE index, Lisp currently sits at #32 on their ranking of programming language popularity. (Common Lisp specifically is somewhere between 51 and 100.) You can certainly imagine that Lisp game programmers are an even smaller group. And then when you consider non-trivial games, the games made by teams, the ones that are significant enough in scope to potentially fund the studio building them… well, I don’t think it’s much of a leap to say that you can probably count the number of those games on one hand.

Suffice it to say that not a whole lot has been written on the art of developing games in Lisp, relative to other languages. As a leader of one of the aforementioned five-or-fewer teams building a sizeable Lisp game (Common Lisp, in our case), I want to contribute to that collective knowledge base. But first, a caveat: It would be premature to consider either (defun games ()), as a studio, or myself, personally, as any kind of “authoritative voice” in the Lisp game community. Spycursion is the first game that we as a studio are producing, and it hasn’t even been funded yet. (Hint hint, Kickstarter, Feb. 26, hint hint!) And although I was a hobbyist Common Lisp developer for about seven years before starting work on Spycursion last year, there’s still a lot about this language I don’t know, and there are certainly people who could write better/faster Lisp game code than me… I’m just the only one who was insane enough to start this project. 😉 We also recently picked up another developer (he hasn’t introduced himself yet, but he will) whose opinions may differ from mine. Taken altogether, what I’m trying to say is: I’m not a guru. I’m just one guy trying to find his way on this bizarre-yet-beautiful landscape, just like everyone else!

Aaaanyway, Lisp games. The first thing you should know, before you go tell all your friends “Hey, they wrote Spycursion in Lisp, let’s create our Overwatch/Pokemon/Dark Souls cross-over in Lisp, too!” is that Spycursion is weird. The game itself is weird, its architecture is weird, its developers are weird… Point is, we’re working with some parameters that make Lisp not only suitable for Spycursion, but perhaps even the ideal language for it. Your Overwatch/Pokemon/Dark Souls cross-over may not fit the bill. (Or it may. I’ll let you know after we release Game #2.) It’s also important to remember that Spycursion has a client and a server component, both of which are written in CL. This was not always the case for the client; I’ll share that story in a minute. But let’s talk about the server first.

As we’ve mentioned before, Spycursion has its own programming language. You may have read about how domain-specific languages (DSL’s) in Lisp are grand, how they’ll make your coffee for you, blah blah blah. In my experience, a lot of the conversation about this can get pretty academic, but here’s a summary of what happens when you compile and run a Slang program in the world of Spycursion:

1. Compilation: The server parses your program (assuming it is syntactically correct) and creates a tree of Lisp forms from it.
2. Runtime: Some contextualization happens with that tree of Lisp forms, and then it is evaluated within the context of an in-game device.
3. (Optional): If the program needs to communicate with another program on another in-game device, via in-game network, then some magic happens to determine where that program is, and runs it, too.

This is of course a lot more complicated than I’m making it sound, but it’s still far less complicated than in probably most, if not all, other mainstream languages. Lisp allows you, as a player, to run your own code within someone else’s online game. (And yes, we do run security checks on your code before we parse it. We’re not n00bs, thankyouverymuch.) This is not merely simulated. There are plenty of places within Spycursion where we “cheat” to make things easier on ourselves — hey, we’re an indie, cut us some slack — but Slang is not one of those places. Just thinking about trying to do all of this in C++ gives me an icky feeling. (I eagerly await the e-mail from someone telling me how you can do it in C++, if you just jump through these 27 hoops…)

So, yes, Lisp is awesome as a game server, but I have a feeling that’s not what most of you are here for. You want to know why we’re using Lisp for the Spycursion client, and how you can write your own amazing thing in Lisp too, and clearly the answers are because I’m insane and you should stop listening to me because it’s fun, dadgum it, and by being a masochist early adopter and maybe having a fairly simple game, graphically. Sarcasm aside, these questions deserve longer answers:

Why Lisp (for the client)? Okay, story time. Long, long ago, when the Spycursion server was just Slang and a WebSocket interface and not much else, I set about creating a text-only testing client. I did this in Python, figuring it would be relatively quick and easy, which it was. At this point I’d not completely decided on how the client would be implemented, but I reasoned that, despite Python not being my best language, it would be easy to find developers for — so I could handle doing all the server-side Lisp as long as I had someone else dedicated to the client.

But there was a problem: Writing in Python just wasn’t fun. Some of you may not understand this, but it’s a phenomenon I’ve heard about before — once you get “spoiled” by Lisp, you have trouble getting motivated to code in any other “lesser” language. This was before we existed as a studio, when I was doing this project entirely by myself, so I didn’t have anyone else to shove the Python work off to, but I knew it was important to get something visual produced so that I could begin to show people (not simply tell them) what Spycursion was about.

So I chucked the Python client, rewrote the thing in CL, and around that time, started getting ideas of what I felt this fledgling studio could be. In my opinion, the reasons “nobody programs in Lisp” mostly boil down to, well… because “nobody programs in Lisp.” The community is small, so there are few tools available, so the community doesn’t grow because people don’t use Lisp for major projects due to a lack of tools. This is even more true for Lisp games specifically, but I believe that this chicken-and-egg problem can be helped by creating a well-known game in Lisp, if only to point to it as a success and prove that it can be done. Additionally, by open-sourcing the client code (which we’ve not done yet, mostly due to legal questions that need answering), we show new and aspiring Lispers how it can be done. Again, to avoid getting ahead of ourselves — this thing could still fail, perhaps in spectacular fashion. But at least if it does, you’ll get a front-row seat. 😉 And I guarantee you that it won’t be because of the programming language we chose.

Why do you think it will work? As I mentioned before, Spycursion is weird. There are very experienced people within the Lisp community who think that Lisp, in its current state, is not ready for serious game programming. I have no basis on which to disagree with them, but I do think it depends on the type of game you’re creating. Spycursion has several things going for it that make it different from what you might imagine as a “typical game,” especially a AAA game:

  • There’s no twitch action. An ill-timed garbage collection isn’t going to be the difference between life and death. As of yet, we’ve made no attempt to customize GC or really push for low-level performance in ways that might be challenging to do in Lisp. We might, at some point, but I’m guessing we won’t even need to.
  • Much of the tricky stuff is offloaded to the server. Don’t underestimate the advantage of having a server (read: one single platform, of your choice) running parts of your game. That means fewer shared library headaches and fewer client performance woes — assuming you do everything asynchronously — for starters. (Oh, but please don’t be that developer who forces a network connection for your single-player game. That’s just rude.)
  • We don’t need to build for consoles. Ever tried to run Lisp on the Switch? I haven’t… and I don’t want to.
  • Physics? We don’t need no stinkin’ physics. Or at least no physics that can’t be calculated based solely on what I remember from 9th grade Geometry.
  • We keep graphics simple. I don’t really know how to quantify that, but the game is meant to be played 3rd-person, from a distanced zoom level. (We let you zoom in, but I wouldn’t necessarily recommend it.) The assets will be relatively low-poly and there will be relatively few of them. Note that “simple” does not mean ugly. We’re aiming to make it all look good while still putting most of our effort into gameplay — graphics are one of the places we “cheat.” You’ll be able to judge how we’re doing once we finish the Kickstarter trailer.

Put all of the above together, and you get a picture of a fairly minimalistic game client that really gets out of the way and lets you enjoy playing. (Or at least that’s the picture I hope you get!) And to be clear, I am not saying that fancier, AAA-level games can’t be made with Lisp. I’m saying I really don’t know, but this is our attempt to get as close as we can with the resources we have.

How are you doing it? I would say “pain, sweat, and toil,” but that kind of describes all game development, so…

  • The engine. Oh, just kidding, we aren’t using one. In 2018, I know of at least three Common Lisp 3D game engines in varying stages of development: First Light, Trial, and cl-bodge. In 2017, when I was researching these engines, they were all either too early in development or otherwise just didn’t fit our needs. They might now, but I haven’t followed their progress enough to know for sure. This brings me to what is maybe the most important thing to know if you’re looking to write a non-trivial Lisp game in 2018: You have to be okay with writing a lot of things yourself. Lisp makes this fun to do, and might still even save you time overall depending on just how much you need to write. But it’s a very different mindset from being, say, a plug-and-play JavaScript developer.
  • CEPL. CEPL is an abstraction layer around OpenGL which has been a joy to work with, once I figured out how. (Tip: See the author’s video stream, it’s great.) Its goal seems to be to make OpenGL lispy and fun to work with, and I really think it has achieved that. Spycursion uses CEPL extensively, and while I’m sure we’re doing some things suboptimally, performance has been fine so far. We’re also using cepl.sdl2 which brings in cl-sdl2.
  • QTools. QTools is a lispy wrapper around Qt 4. Most games probably won’t want the heft of Qt, but our device UI actually needs something a bit more featureful, so it’s worth mentioning. We have run into some build problems with QTools, particularly on Linux. Add in that there doesn’t seem to be a Qt5 update on the horizon, and we may well end up switching away from it (to what, I can’t say yet). But it’s currently working well, for what we need it for.
  • Other libraries

And that’s it! Lisp game development isn’t for everyone, but I stand by my assertion that it can be done, and done well, given the right game — and perhaps the right amount of patience.

TL;DR:

  • Use CEPL. Or one of the engines, if they fit. Or maybe both.
  • Yes, you should do it, even if you fail. That’s how the community grows.
  • Be prepared to write a lot of stuff yourself. This is not necessarily a bad thing. It’s fun!
    • Except skeletal animation. Don’t write that yourself.
  • Support our Kickstarter starting on February 26th!



Post-PAX Plans: What’s Next?

PAX was an amazing experience for us and we’re incredibly happy with how it turned out. We want to thank everyone who got in involved, no matter how big or small your contribution. Even if you just retweeted one of our PAX tweets, you’ve helped us and that means a lot. We’re extremely grateful for your support.

What’s next for (defun games ()) and Spycursion you ask? Well, let’s run you through it.

Currently, we’re deep in development on a whole bunch of game developer stuff. Animations, bug squashing, camera code, etc. Nothing super exciting or announcement worthy just yet. We’ll be sure to update you on anything exciting!

We do have a few other things planned, though. Firstly, we’re getting ready to launch our Discord server. This will be a great place for people to discuss Spycursion, us, the weather, or whatever else strikes your fancy. More details on that coming soon.

Secondly, since our first puzzle event was such a success, we’re thinking of doing another! If you took part in the PAX West puzzle event a couple of weeks ago then stay tuned because we’ll have more to say about this soon. If you have any feedback about our last puzzle event, please don’t hesitate to hit us up on Twitter, email, or Reddit.

And, of course, we are continuing to plan the various aspects of our Kickstarter campaign. We want it to be as good as it can possibly be so don’t hold your breath for a launch date just yet. It’ll begin when it’s ready and when we’re ready.

Join the Team

As always, we’re looking for lovely people to join our team and lend a hand. If you, or someone you know, is into the idea of Spycursion and has some expertise in software development, particularly experience with Common Lisp or a Lisp-like language, then get in touch. We’re extremely passionate about Spycursion and, like you, we want to see it come to life as soon as possible. A little help on the development side will allow us to bring that dream closer.

We’d love to hear from you! For more information, see here.