The basics of game programming in Common Lisp.

Common lisp is a wonderful language, a lot of people tend to think it is a functional programming language, and while that is not a bad thing, Lisp is not a functional programming language, it is a multi-paradigm programming language. While it is not considered good Lisp style, you can program in an imperative way; Why am I telling you all this? well, because people always object that being functional makes it difficult to program games, which is not completely true either, but Common Lisp will work with you and not force you to work around it.

While I am in the theme of fallacies, another false misconception about Lisp is that it does not have libraries, it does have more than a descent amount of native libraries, Common Lisp implementations also have FFIs which has allowed to create bindings to very useful C libraries, and since bindings are commonly not “Lispy” some times we have great adventurers creating wrappers that made them feel like they were written for Common Lisp.

So, What do you need to program a game?, in the simplest terms you need:

  • Output: A place to show text, graphics, lights, sound or whatever your game uses for interaction with the user.
  • Input: Devices to interact with the game (i.e, move the character, move icons, write text, etc).
  • Assets: Different types of things that go into your game (images, models, sounds, maps, music).
  • Constraints: A set of rules that physically or logically constraint the actions that you as a player can do (i.e. physic engines)
  • Logic: A set of rules that affect how the user interacts with elements in the world. (i.e. open a chest, a door, use a computer etc)

    You might argue there are more things, but to me this five make the core of what you need. So lets break this down for Spycursion, which is our game, I will intentionally leave out the socket magic happening between the client and server, which is necessary in any MMO.

    There are two types of interaction in our game, the Computer GUI and Terminals and the 3D world.

    Spycursion Computers

    Input/Output

    The computers in Spycursion are meant to look like real computers, once you start using a computer you will either see a GUI environment or a Terminal, so what we need here is a GUI library, we could (and actually thought about) write one from scratch, but with so many solutions in existence out there we decided to use one with the characteristics we needed, and the winner was Qt, one of it’s redeeming qualities being portability, which is important when you are writing a game that runs in Linux, Mac and Windows. To write our Qt in a lispy way we use the fantastic Qtools library written by Nicholas Hafner (a.k.a Shinmera).

Assets

Qt provides the necessary tools to import icons, images, play sounds, and even render html, so it is relatively easy to get a full screen application that looks like a computer desktop.

Constraints

Constraining a GUI is very easy, you just provide the programs that you want people to be able to use, and what those programs are capable of doing. Constraining a terminal is even easier, you just decide what commands exist and how they interact with others (piping, I/O). In both cases you can restrict user rights, so it ends up being a bit like real life system administration, but with more control.

 Logic

Implementing Logic in Qt is very straight forward, you can decide what happens when you press a button, hit enter, change text, etc.

I won’t provide examples because there are plenty on the Internet.

Spycursion 3D World

Output

Since we are shooting for a 3D world the safest bet is to use OpenGL, Lisp has the cl-opengl library, but that is too verbose, the good news is that Chris Bagley (a.k.a Baggers) created CEPL, which has as a design goal, and I cite “… making the user feel that GPU programming has always been part of the languages standard.”, asides from that promise, Baggers has spend hours doing great videos which he has shared on Youtube.
To run OpenGL you need what is called a surface, which is basically a graphic container, when we started doing this the only stable option for CEPL was SDL2, so we to use cl-sdl2 too. The library, in this setup, is used only to have a display window, but it is capable of a lot more, as an example you can look at sketch, which you could use to create 2D games.

Input

A lot of the things we care about were also concerns to Baggers, and he has a whole ecosystem of related stuff, some of which even come installed with CEPL or are part of it, for Input his contribution is called skitter, which allows us to detect mouse movement, position, clicks, keyboard presses and all that jazz, it can even handle game controllers. While we use it with CEPL it is independent of it.

Assets

So far we have been worried more about showing a 3D world, some menus and text, so I will talk only about those aspects, of course there are options to import and play sound, but I will focus on 3D models. You should be able to use a 3D modeling program to create your assets, save files, and get the data from those files imported into your program, that is the job of the great assimp library, and of course we have classimp which is classier ;-), with that library you can import several 3D model file formats, so you can use a variety of programs to create your model, including free options like Blender, or you can buy them in a format you can use. Of course this only creates a data structure, so you have to do all the wiring yourselves, but you will be able to import models including the animations which you need for your game.

Constraints/Logic

All constrains and logic are programmed in Lisp, with some help from stenciling, this is one of the few instances where we used cl-opengl directly instead of using the CEPL scaffolding. In general Common Lisp is a very potent programming language, so writing logic in it is very straightforward and flexible. We are currently not using physics engines but there are libraries for that too.

Some final thoughts

There are many wonderful tools you can use to make your game, we are just talking about the ones we used, you could use Xelf for 2D games, or Trial for 3D and 2D, there are libraries for text rendering both in 2D and 3D, libraries to do text based games (i.e. cl-ncurses), tools for importing and playing sound. There have been games written in Common Lisp in recent years, and there are Common Lisp game jams (the next one starts on April 18th, 2019), the fact is that writing games has been improving the ecosystem.

If you ask for it, we could expand in some of these themes, but the basics are the same, choose the type of game you want and the look for it, choose the adequate input and output libraries, add suitable libraries for importing assets, use some libraries for collision detection and physics if necessary, add your own logic, mix everything with tons of creativity, and you’ll have a game.



Some things we’ve learned developing a 3D game for Lisp.

Hi, Mauricio Here,

This blog post is kind of a developers blog, as I said in my only previous post I’ve been dealing with the MS Windows side of things, I also made some 3D models using Blender, and use the abstractions Scott had created to create a a new scene for the game (a bedroom), but of course my job has not been limited to that. We’ve learned some stuff that we would like to share, so that is what I am doing.

Classimp on Windows.

One of the earlier lessons we learned was related to Assimp, for one thing downloading the binaries and moving the dll to your project directory does not work, in recent Assimp versions the name of the dll isn’t even one of the names searched by Classimp, and renaming or making the code match the dll name does not do the trick. In order to make classimp work we had to compile with gcc instead of visual c, using mingw,, this produces a libassimp.dll which is what classimp looks for and that DLL works fine. The latest Assimp library supported by Classimp is 3.3.1, so if you want to compile it yourself download that version and then follow the instructions here.

When you compile with MingGW using the instructions in the link the DLL will be in the “assimp-3.3.1\build\tools\assimp_cmd” directory, and asides from that you get an assimp.exe executable that is very useful for asset conversion. Currently we are consuming fbx files, so when I created models in Blender I exported them as fbx files, and that caused problems in some cases, so we adopted a different  workflow which so far guarantees a model that loads correctly in CEPL, we use assimp.exe in the command line to convert the .blend to .dae which is the extension for Collada, an open exchange format, and then use Autodesk FBX Converter to convert the .dae to .fbx. This workflow gives you better control over the way that the files are converted.

Why not reading the .blend files directly?, Well for one thing we had some pre-made assets and those were fbx, and we want to be uniform., another reason is that sometimes the file would not be read correctly by classimp, but assimp.exe tries several stuff and is able to export it, as long as you remove any lamps form the scene (I also remove the cameras for good measure). And a third reason, is that sometimes we want to pre-transform the vertices  specially with several objects in the same file, and doing this during the export simplifies the code to read the file. So we do use this:

assimp export file.blend  file.dae -ptv

In the command line, where -ptv stands for pre-transform vertices, and assimp guesses the output file format from the extension of the second file. Assimp does not export fbx , that is why we export to ,dae and then convert. Classimp can read .dae files correctly, but fbx files are smaller, and the difference in load time is negligible for our purposes, and as I said we want to keep uniformity.

Another good tip is to use the assimp viewer to open an asset, and then look at the log file, this will warn you of possible problems when importing with classimp.

Multiple Textures in one Mesh.

We ran into a problem with a model of an avatar we were using in the game, the textures were not imported correctly, the head texture was applied to the body, so the model looked like it had no pants, and it had some weird things in the skin (actually stretched head features). The problem is that when you run (classimp:import-into-lisp) each mesh object contains only one texture. however the model had a mesh with two textures.

Luckily Blender allows you to select the portion of the mesh that is assigned to a particular texture and separate it. So we separated the head from the body, re exported, re-sorted the textures and voila, problem solved.

How do you know the order of the meshes? Well, the first thing is to name things right in Blender, then the structure you get when you import into lisp has a tree and that gives you the clue.

SPYCURSION-CLIENT> (ai:import-into-lisp (car *char8-data*))
#<CLASSIMP:SCENE {100A0E4AE3}>
SPYCURSION-CLIENT> (ai:root-node *)
#<CLASSIMP:NODE {1004FA9443}>
SPYCURSION-CLIENT> (ai:children *)
#(#<CLASSIMP:NODE {1004FA8663}> #<CLASSIMP:NODE {1004FA8903}>
  #<CLASSIMP:NODE {1004FA8B93}> #<CLASSIMP:NODE {1004FA8E33}>
  #<CLASSIMP:NODE {1004FA90D3}> #<CLASSIMP:NODE {1004FA9373}>)
SPYCURSION-CLIENT> (loop for x across * collect (ai:name x))
("Head" "Gadget" "Bag" "Body" "Jacket" "Hair")
SPYCURSION-CLIENT>

So the first mesh is the head, the second the Gadget, and so on. The pre existing textures worked well because I just separated and saved again so the UV coordinates remain intact,  now each mesh reads from the right image, and maps the right coordinates.

Blender

One of the models I was making was a full room, to do it I used Archimesh, the room was simple, a rectangular room with a window and a door,  and some cabinets, this worked out perfectly in Blender, but not so much when I imported the models in CEPL. The thing is that boolean operators produce weird geometry, and the holes in the walls for the door and window are produced by boolean subtraction. In other models the sub surface modifiers looked like crap when imported,

So I decided to re-make the room the old fashioned way, loop-cuts and face deletion, replace the old room, and apply all of the modifiers to other objects before exporting, that way the look was predictable.

The other thing I learned is that you need to either use the a processing flag in classimp or export the model with pre-transformed vertices, otherwise the objects in the scene were not placed in the right position. I used Blender to lay out the room furniture, and then exported each element separately, that made it easier to place and remove things, and handle the textures.

I also spend some time baking textures, and exporting the UVs, the UVs exported from Blender are immediately usable, and you can use DIRT to import the texture, so asides from the tedious process of UV wrapping the objects, assigning the image, and baking the textures into it, the process is very painless.

Final Thoughts

After ironing out the exporting problems saving models so they can be used with CEPL turns out to be a straight away process, there are some things in our code we are optimizing to make the process of maintaining the software and adding stuff more straight forward, but we have a working skeleton,  we can create rooms and cities, place avatars, animate them,  etc.

Of course there is still work to do, and Spycursion is curious in that it has two environments, the OS and the 3D world, but even when we some times hit ourselves against a wall, there is always a door a couple of steps away, as we learn more is like turning the lights on, we now where the walls doors, windows and other stuff is, and that allows us to move freely.



Introducing myself (Mauricio)

Mauricio's Picture :-)
Don’t be scared is just my face

Hi, as Scott said, I have not introduced myself, my name is Mauricio Fernández, I’m a gamer since computer games exists, and I have been interested in programming since around 1984, when I learned Basic for the VIC-20, some of my first programs were simple games, since then I’ve kept an interest in games, so I know some about it, even without having been part of the game industry yet.  I’ve been programming since the 80s, and it is one of the things I enjoy the most.

Some of the people reading (defun games ()) blog posts might recognize my reddit handler (maufdez), I have been active there for a while, since I found the subreddits a little after I began learning Lisp. I also wrote a blog, which I have abandoned for a while now (if you are a follower, sorry about that).

On his last blog post Scott talked about his thoughts about game development in Common Lisp. I’ve been working on the MS Windows side of things, among other things, so I thought I could complement with some of my own thoughts.

About Portability

While most hackers try to live away from MS Windows, selling a Linux only game is a bit like writing a book in Icelandic, you are limiting your market from the get go (of course there are more Linux users, than citizens in Iceland). But what this means is that we want to write the client to run on Windows 10 too.

In my experience, if you write a program in portable Common Lisp, there is a very high probability that it will run on any platform that has a ANSI compliant Common Lisp compiler. The problem is not Common Lisp, you start getting into issues when using libraries that depend on a Foreign language, specially in windows where these depend on DLLs. So you say, that proves it, Common Lisp is not mature enough for game development, but the truth is that a lot of the problems you get are also present in other programming languages, the proof is that you can find questions about the same problems in forums dedicated to Python, Java, and even C, C++, and others.

I tend to try to avoid FFI libraries whenever I can, but unless we are willing to reinvent all kinds of wheels, we have to use things like sdl2, qt (or similar), OpenGl, assimp, and others, and that sometimes means that your program will fail in the foreign call, or loading the foreign library (which may fail because a third DLL that you don’t know about).

The problems with foreign libraries are not limited to Windows, you might run into similar problems going from a Linux distro to another, and , as I said, in most cases the problem is not a Common Lisp problem, and developers working in other languages run into the same issues.

The good news is that we know that these libraries worked on some C or C++ project, with the right SOs or DLLs, so it should most likely also run on Common Lisp, and that is what we have found so far. Which means that, with perseverance, we can get our game working, and learn how to smooth all the wrinkles, and that is part of the value we get from this, “Game 2” should be easier to develop once we know where the bumps on the road were when creating “Game 1”, and so forth.

A lot of the libraries we are using are a labor of love. As far as I know Baggers does not receive any remuneration for developing CEPL, and I think neither does Shinmera for working on Qtools, to mention just a couple, some projects are more complex than others, going from simple bindings, to completely lispified ways of writing code, so the parts are there, we have to piece them together, document our findings, and help to improve the existing tools however we can.

I think that making games, will improve the tools, which will make it easier to produce more games, which in turn will show us more ways of improving the tools, and that continuous spiral will make Lisp as good or better than other languages in terms of game development, and I think we have enough people writing (or trying to write) games to make this happen.

Releated Links

Reddit: u/maufdez
Twitter: @MauricioFdezF
Masstodon: @maufdez@fosstodon.org
Github: maufdez
Blog: Funlisp
Google+: MauricioFernandezF