Anomaly Report #73 (CASSI 001)

From: Burdinjaun
To:                    

Subject: [TS//SCI] United States Armed Forces, Cyber Directorate — Possible Security Breach
Status: Investigation in Progress
Interviewer:                         , Military Investigator

Verifying… [100%]

Deciphering… [100%]

 


 

Interviewer: Please verbally express your consent to record this interview.

Tomás Lednura, PhD: I consent.

Interviewer: Can you summarize the facts of what happened on Oct. 24th, in the Annex B Conference Room 109, at 13:34?

Lednura: OK, as I recall it… Hmm, 17 people were summoned to the room due to an urgent security breach. 13 were there when the meeting started and the doors were closed. Four were high ranking officers, six were cybersecurity experts. The rest were CASSI developers.

Interviewer: For the record, CASSI is the Computer-Assisted Security Surveillance Intelligence. Do you recall all the names?

Lednura: You should have them in the written report I sent… (trails off) Jack from cyber security, my boss, called the meeting. I made the discovery of the information leak, and he restated what I had told him. At 13:01, one of the servers hosting CASSI started extracting data from the other servers and sending it to external sources. There were no known intrusions at the time, as verified by extensive malware checks. The data transfer processes were not spawned from a cron job or any other server configurations, and resisted all attempts to kill them. After getting everybody up to date on the situation, an engineer from the AI development team gave an assessment… (pause) Lisa, I think. Anyhow, she said that CASSI was only capable of monitoring communications and looking for possible national security threats. Any alarms would be relayed to a GUI in a room where some analyst would continue the work, and the system had no volition or any I/O capable of moving files or sending messages to the outside. I guess she could give you a better summary.

Interviewer: Sadly, we can’t ask her, nor anyone else. After the events of that day, one by one everyone in that room has died or disappeared. Except you, Dr. Lednura.

Lednura: What? How is that possible? I talked to Glenn a week ago!

Interviewer: Please stay on topic. We’ll talk about Mr. Lewis’ disappearance later. We need all the facts first.

Lednura: (sighs) I think those are all the facts any of us had prior to the meeting. During the meeting, we found that large amounts of data were being transferred. There was no rhyme or reason as far as security classification, but most of it was generated by CASSI processes. Like I said, every time we killed one of those processes a new one replaced it, and no other countermeasures worked, either. Gen. Miller asked for a summary; we told him what we knew and showed some evidence illustrating what was happening. That’s when I suggested taking the server offline, but there was a lot of debate about that. The CASSI guys were adamant that there was no way it could be responsible for the situation.

Interviewer: Was it?

Lednura: The evidence was inconclusive, but it was strong enough to suggest that turning off the server could stop the leak. I explained that, and Gen. Miller gave the OK to shut it down, even with the devs protesting about its state being corrupted, losing generations and all of that. As soon as he gave that order, the alarm went off, and the monitors showed the activation of something called Protocol Z14. I didn’t even know what Protocol Z14 was, I still don’t. But I distinctly remember hearing the big doors locking.

Interviewer: The big doors?

Lednura: That’s internal jargon for a complete lockdown, basically. Gen. Miller gave us this confused look and said that Protocol Z14 could not be activated without his OK. But we were all locked out of our systems.

Interviewer: Aside from that, did he reveal anything about the nature of the Z14 protocol?

Lednura: No, we’re all trained to avoid asking too many questions. But a voice recording sounding with the alarm said we had five minutes before the whole floor was incinerated, so we were frantically trying to override the security systems preventing us from getting out, or at least the ignition systems about to burn us alive. (sigh) But it was useless. After a few minutes we gave up and accepted our fate.

Interviewer: Just like that?

Lednura: What else could we do? We were in a concrete basement, with two-meter reinforced walls, a single exit sealed by a solid steel door, no working overrides, and a (expletive) incendiary bomb about to go off!

Interviewer: OK, and-

Lednura: Gen. Miller was starting to say something when the alarms ceased, and a green notification on the monitors signaled that Protocol Z14 had been suspended. Everybody was cheering, but the general just had that same confused or worried look on his face.

Interviewer: I can tell you this much: Protocol Z14 was designed to not be stopped, paused or suspended in any way. Once initiated, there is no way to stop it, by design.

Lednura: I guess that’s why he asked right afterward who had stopped it. We were all as puzzled as he was. Then a bunch of code appeared on my screen-

Interviewer: That’s enough, thank you. Now let’s talk about Mr. Lewis’ disappearance.

 

(click sound; end of recording)



Project Update for June 2019

Common Lisp: How deep does the rabbit hole go?

Being that both Scott and I had to go back to getting jobs to put bread in the table things are going slower than when we were both 100% dedicated to the project. In the last update Scott mentioned some ideas we had around re-structuring the client, as you may imagine we can manage to keep the server side purely in Common Lisp, but the client side is a bit different, we need to create UI elements, and 3D graphics, as well as adding assets. We talked about moving away from Qt, not because Qtools are not capable, but mainly because in it’s current state UI elements cannot be rendered in the OpenGL window, which creates a disjointed experience for the user.

So we started an internal discussion to evaluate the future of Spycursion’s development, things that we wanted, things that we needed contrasted to what we have and what we need to implement, and how home grown that implementation needs to be, of course we want to use existing solutions where it makes sense, but we have our own philosophy around development.

Quick story, we considered going down the path of using Godot, using gdnative to be able to program all of the game logic in Common Lisp while using the game engine capabilities to import assets, create the world, apply physics, etc. I even started the task of creating the bindings1 using CFFI. All of this seemed like a good contribution to the lisp game developers community, and there are plenty of Godot tutorials on line. So asides from having to redo everything in Godot, what stopped us?, it is not Lisp far enough down.

If we make it so we can program Godot’s game logic in Common Lisp we are only layering a coat of usability on top of an otherwise C++ stack. Lisp Hackers could only contribute to the engine through plug ins or functions that work with the API, but if they want to go deeper they have to switch languages. So the fair question now is, how far down we can go in our current stack before we have to switch languages?, after all a lot of what we do in game development is done through FFI. The answer is, arguably, farther down than if we just wrap Godot, of course with some trade-offs. So what will Godot do for us, here is a non comprehensive list:

  • Asset management, sounds, music, sprites/models.
  • A visual environment to create levels and assign properties to in-game objects (walls, ground, etc).
  • A physics engine.
  • Custom Shaders (with their own language).
  • GUI which can be layered on top of the game.
  • A simple scripting language for the logic.

What we have now with our prototype client:

  • Asset management, mostly 3D models only, character animations are rendered correctly, 2D sprites can be imported as planes with texture.
  • A non visual environment to create complex scenes.
  • Custom Shaders using the excellent CEPL library created by Chris Bagley (akaBaggers), shaders are written in Vari which is very Lispy.
  • The complete Common Lisp language for the logic.

The Missig parts

Asset management

We have not added any sound to our game yet, so we haven’t dealt with importing and managing sound including music, however there are several options out there like Pavel Korolev’s (aka Borodust) bodge-openal, or Nicholas Haffner’s (aka Shinmera) harmony, and some others. So we have native options we can integrate in our client with relative ease.

Scene management

Although I listed the “non visual environment” I guess I should mention two things. First, in our current work flow I was able to create a scene in Blender directly, and then import the elements one by one in the SC client replicating the placement from blender automatically. So in a way we have a visual environment to create the scene. I think however, that using CEPL it should be simple enough to create a solution where you move and rotate things around as needed. So far we haven’t got a need for that.

UI

We are working currently working on integrating bodge-ui (suggested by our good friend Borodust), which will allow us to layer UI elements on top of the game, this is what will substitute Qtools, I’ve made some basic testing and the UI renders well in Linux and Windows, I can’t check Mac but we will get around to testing more thoroughly once we integrate things in the Spycursion client. We also talked about having the users be able to create “web pages”, the solution in Godot seems to be using a combination of bbcode and the GUI, which got us thinking, we don’t really need a full blown HTML render in game, in fact is safer to use something like bbcode but with the addition of forms.

Is there need for a Common-Lisp game engine?

We have been constructing a game, not a game engine, using loose parts created by great programmers, by programming the code to be modular and reusable we get a similar effect to having a full blown engine, as an example. if we need to add another scene we just create a scene file, place the assets in a convenient place, and declare things and we have a scene ready to be rendered. I talked about Shinmera’s harmony, and Borodust’s bodge-ui, these wonderful programmers are already giving us the building blocks, so you can make your own mix and match, or you can choose to use their full solutions i.e. Radiance, CL-Bodge, etc. We even thought about going 2D with Dave O’Toole’s (aka dto) Xelf.

Ongoing work

There is a lot going, very slowly but going, we are refactoring the client to make the code modular an easy to reuse, some manual steps are being moved to CLOS methods (before, after and around), some hard coded values are being moved to special variables, etc. The code has to be easy to maintain, and our initial approach needed some refining. I’ve been working mainly on the client, and Scott more on the server, and a lot of what we have been doing is “invisible”, but we are doing what must be done. Hopefully as we add/refine the visual aspects of the game we will be able show the progress instead of just telling you about it.

Patreon

We want to publicly say thanks to our Patreon supporters, although we cannot live from their contribution some of the game expenses will be alleviated, and it shows us that there is still interest on having this game become a reality.

Footnotes

1. Borodust already did Godot’s bindings



The basics of game programming in Common Lisp.

Common lisp is a wonderful language, a lot of people tend to think it is a functional programming language, and while that is not a bad thing, Lisp is not a functional programming language, it is a multi-paradigm programming language. While it is not considered good Lisp style, you can program in an imperative way; Why am I telling you all this? well, because people always object that being functional makes it difficult to program games, which is not completely true either, but Common Lisp will work with you and not force you to work around it.

While I am in the theme of fallacies, another false misconception about Lisp is that it does not have libraries, it does have more than a descent amount of native libraries, Common Lisp implementations also have FFIs which has allowed to create bindings to very useful C libraries, and since bindings are commonly not “Lispy” some times we have great adventurers creating wrappers that made them feel like they were written for Common Lisp.

So, What do you need to program a game?, in the simplest terms you need:

  • Output: A place to show text, graphics, lights, sound or whatever your game uses for interaction with the user.
  • Input: Devices to interact with the game (i.e, move the character, move icons, write text, etc).
  • Assets: Different types of things that go into your game (images, models, sounds, maps, music).
  • Constraints: A set of rules that physically or logically constraint the actions that you as a player can do (i.e. physic engines)
  • Logic: A set of rules that affect how the user interacts with elements in the world. (i.e. open a chest, a door, use a computer etc)

    You might argue there are more things, but to me this five make the core of what you need. So lets break this down for Spycursion, which is our game, I will intentionally leave out the socket magic happening between the client and server, which is necessary in any MMO.

    There are two types of interaction in our game, the Computer GUI and Terminals and the 3D world.

    Spycursion Computers

    Input/Output

    The computers in Spycursion are meant to look like real computers, once you start using a computer you will either see a GUI environment or a Terminal, so what we need here is a GUI library, we could (and actually thought about) write one from scratch, but with so many solutions in existence out there we decided to use one with the characteristics we needed, and the winner was Qt, one of it’s redeeming qualities being portability, which is important when you are writing a game that runs in Linux, Mac and Windows. To write our Qt in a lispy way we use the fantastic Qtools library written by Nicholas Hafner (a.k.a Shinmera).

Assets

Qt provides the necessary tools to import icons, images, play sounds, and even render html, so it is relatively easy to get a full screen application that looks like a computer desktop.

Constraints

Constraining a GUI is very easy, you just provide the programs that you want people to be able to use, and what those programs are capable of doing. Constraining a terminal is even easier, you just decide what commands exist and how they interact with others (piping, I/O). In both cases you can restrict user rights, so it ends up being a bit like real life system administration, but with more control.

 Logic

Implementing Logic in Qt is very straight forward, you can decide what happens when you press a button, hit enter, change text, etc.

I won’t provide examples because there are plenty on the Internet.

Spycursion 3D World

Output

Since we are shooting for a 3D world the safest bet is to use OpenGL, Lisp has the cl-opengl library, but that is too verbose, the good news is that Chris Bagley (a.k.a Baggers) created CEPL, which has as a design goal, and I cite “… making the user feel that GPU programming has always been part of the languages standard.”, asides from that promise, Baggers has spend hours doing great videos which he has shared on Youtube.
To run OpenGL you need what is called a surface, which is basically a graphic container, when we started doing this the only stable option for CEPL was SDL2, so we to use cl-sdl2 too. The library, in this setup, is used only to have a display window, but it is capable of a lot more, as an example you can look at sketch, which you could use to create 2D games.

Input

A lot of the things we care about were also concerns to Baggers, and he has a whole ecosystem of related stuff, some of which even come installed with CEPL or are part of it, for Input his contribution is called skitter, which allows us to detect mouse movement, position, clicks, keyboard presses and all that jazz, it can even handle game controllers. While we use it with CEPL it is independent of it.

Assets

So far we have been worried more about showing a 3D world, some menus and text, so I will talk only about those aspects, of course there are options to import and play sound, but I will focus on 3D models. You should be able to use a 3D modeling program to create your assets, save files, and get the data from those files imported into your program, that is the job of the great assimp library, and of course we have classimp which is classier ;-), with that library you can import several 3D model file formats, so you can use a variety of programs to create your model, including free options like Blender, or you can buy them in a format you can use. Of course this only creates a data structure, so you have to do all the wiring yourselves, but you will be able to import models including the animations which you need for your game.

Constraints/Logic

All constrains and logic are programmed in Lisp, with some help from stenciling, this is one of the few instances where we used cl-opengl directly instead of using the CEPL scaffolding. In general Common Lisp is a very potent programming language, so writing logic in it is very straightforward and flexible. We are currently not using physics engines but there are libraries for that too.

Some final thoughts

There are many wonderful tools you can use to make your game, we are just talking about the ones we used, you could use Xelf for 2D games, or Trial for 3D and 2D, there are libraries for text rendering both in 2D and 3D, libraries to do text based games (i.e. cl-ncurses), tools for importing and playing sound. There have been games written in Common Lisp in recent years, and there are Common Lisp game jams (the next one starts on April 18th, 2019), the fact is that writing games has been improving the ecosystem.

If you ask for it, we could expand in some of these themes, but the basics are the same, choose the type of game you want and the look for it, choose the adequate input and output libraries, add suitable libraries for importing assets, use some libraries for collision detection and physics if necessary, add your own logic, mix everything with tons of creativity, and you’ll have a game.



Some things we’ve learned developing a 3D game for Lisp.

Hi, Mauricio Here,

This blog post is kind of a developers blog, as I said in my only previous post I’ve been dealing with the MS Windows side of things, I also made some 3D models using Blender, and use the abstractions Scott had created to create a a new scene for the game (a bedroom), but of course my job has not been limited to that. We’ve learned some stuff that we would like to share, so that is what I am doing.

Classimp on Windows.

One of the earlier lessons we learned was related to Assimp, for one thing downloading the binaries and moving the dll to your project directory does not work, in recent Assimp versions the name of the dll isn’t even one of the names searched by Classimp, and renaming or making the code match the dll name does not do the trick. In order to make classimp work we had to compile with gcc instead of visual c, using mingw,, this produces a libassimp.dll which is what classimp looks for and that DLL works fine. The latest Assimp library supported by Classimp is 3.3.1, so if you want to compile it yourself download that version and then follow the instructions here.

When you compile with MingGW using the instructions in the link the DLL will be in the “assimp-3.3.1\build\tools\assimp_cmd” directory, and asides from that you get an assimp.exe executable that is very useful for asset conversion. Currently we are consuming fbx files, so when I created models in Blender I exported them as fbx files, and that caused problems in some cases, so we adopted a different  workflow which so far guarantees a model that loads correctly in CEPL, we use assimp.exe in the command line to convert the .blend to .dae which is the extension for Collada, an open exchange format, and then use Autodesk FBX Converter to convert the .dae to .fbx. This workflow gives you better control over the way that the files are converted.

Why not reading the .blend files directly?, Well for one thing we had some pre-made assets and those were fbx, and we want to be uniform., another reason is that sometimes the file would not be read correctly by classimp, but assimp.exe tries several stuff and is able to export it, as long as you remove any lamps form the scene (I also remove the cameras for good measure). And a third reason, is that sometimes we want to pre-transform the vertices  specially with several objects in the same file, and doing this during the export simplifies the code to read the file. So we do use this:

assimp export file.blend  file.dae -ptv

In the command line, where -ptv stands for pre-transform vertices, and assimp guesses the output file format from the extension of the second file. Assimp does not export fbx , that is why we export to ,dae and then convert. Classimp can read .dae files correctly, but fbx files are smaller, and the difference in load time is negligible for our purposes, and as I said we want to keep uniformity.

Another good tip is to use the assimp viewer to open an asset, and then look at the log file, this will warn you of possible problems when importing with classimp.

Multiple Textures in one Mesh.

We ran into a problem with a model of an avatar we were using in the game, the textures were not imported correctly, the head texture was applied to the body, so the model looked like it had no pants, and it had some weird things in the skin (actually stretched head features). The problem is that when you run (classimp:import-into-lisp) each mesh object contains only one texture. however the model had a mesh with two textures.

Luckily Blender allows you to select the portion of the mesh that is assigned to a particular texture and separate it. So we separated the head from the body, re exported, re-sorted the textures and voila, problem solved.

How do you know the order of the meshes? Well, the first thing is to name things right in Blender, then the structure you get when you import into lisp has a tree and that gives you the clue.

SPYCURSION-CLIENT> (ai:import-into-lisp (car *char8-data*))
#<CLASSIMP:SCENE {100A0E4AE3}>
SPYCURSION-CLIENT> (ai:root-node *)
#<CLASSIMP:NODE {1004FA9443}>
SPYCURSION-CLIENT> (ai:children *)
#(#<CLASSIMP:NODE {1004FA8663}> #<CLASSIMP:NODE {1004FA8903}>
  #<CLASSIMP:NODE {1004FA8B93}> #<CLASSIMP:NODE {1004FA8E33}>
  #<CLASSIMP:NODE {1004FA90D3}> #<CLASSIMP:NODE {1004FA9373}>)
SPYCURSION-CLIENT> (loop for x across * collect (ai:name x))
("Head" "Gadget" "Bag" "Body" "Jacket" "Hair")
SPYCURSION-CLIENT>

So the first mesh is the head, the second the Gadget, and so on. The pre existing textures worked well because I just separated and saved again so the UV coordinates remain intact,  now each mesh reads from the right image, and maps the right coordinates.

Blender

One of the models I was making was a full room, to do it I used Archimesh, the room was simple, a rectangular room with a window and a door,  and some cabinets, this worked out perfectly in Blender, but not so much when I imported the models in CEPL. The thing is that boolean operators produce weird geometry, and the holes in the walls for the door and window are produced by boolean subtraction. In other models the sub surface modifiers looked like crap when imported,

So I decided to re-make the room the old fashioned way, loop-cuts and face deletion, replace the old room, and apply all of the modifiers to other objects before exporting, that way the look was predictable.

The other thing I learned is that you need to either use the a processing flag in classimp or export the model with pre-transformed vertices, otherwise the objects in the scene were not placed in the right position. I used Blender to lay out the room furniture, and then exported each element separately, that made it easier to place and remove things, and handle the textures.

I also spend some time baking textures, and exporting the UVs, the UVs exported from Blender are immediately usable, and you can use DIRT to import the texture, so asides from the tedious process of UV wrapping the objects, assigning the image, and baking the textures into it, the process is very painless.

Final Thoughts

After ironing out the exporting problems saving models so they can be used with CEPL turns out to be a straight away process, there are some things in our code we are optimizing to make the process of maintaining the software and adding stuff more straight forward, but we have a working skeleton,  we can create rooms and cities, place avatars, animate them,  etc.

Of course there is still work to do, and Spycursion is curious in that it has two environments, the OS and the 3D world, but even when we some times hit ourselves against a wall, there is always a door a couple of steps away, as we learn more is like turning the lights on, we now where the walls doors, windows and other stuff is, and that allows us to move freely.



Introducing myself (Mauricio)

Mauricio's Picture :-)
Don’t be scared is just my face

Hi, as Scott said, I have not introduced myself, my name is Mauricio Fernández, I’m a gamer since computer games exists, and I have been interested in programming since around 1984, when I learned Basic for the VIC-20, some of my first programs were simple games, since then I’ve kept an interest in games, so I know some about it, even without having been part of the game industry yet.  I’ve been programming since the 80s, and it is one of the things I enjoy the most.

Some of the people reading (defun games ()) blog posts might recognize my reddit handler (maufdez), I have been active there for a while, since I found the subreddits a little after I began learning Lisp. I also wrote a blog, which I have abandoned for a while now (if you are a follower, sorry about that).

On his last blog post Scott talked about his thoughts about game development in Common Lisp. I’ve been working on the MS Windows side of things, among other things, so I thought I could complement with some of my own thoughts.

About Portability

While most hackers try to live away from MS Windows, selling a Linux only game is a bit like writing a book in Icelandic, you are limiting your market from the get go (of course there are more Linux users, than citizens in Iceland). But what this means is that we want to write the client to run on Windows 10 too.

In my experience, if you write a program in portable Common Lisp, there is a very high probability that it will run on any platform that has a ANSI compliant Common Lisp compiler. The problem is not Common Lisp, you start getting into issues when using libraries that depend on a Foreign language, specially in windows where these depend on DLLs. So you say, that proves it, Common Lisp is not mature enough for game development, but the truth is that a lot of the problems you get are also present in other programming languages, the proof is that you can find questions about the same problems in forums dedicated to Python, Java, and even C, C++, and others.

I tend to try to avoid FFI libraries whenever I can, but unless we are willing to reinvent all kinds of wheels, we have to use things like sdl2, qt (or similar), OpenGl, assimp, and others, and that sometimes means that your program will fail in the foreign call, or loading the foreign library (which may fail because a third DLL that you don’t know about).

The problems with foreign libraries are not limited to Windows, you might run into similar problems going from a Linux distro to another, and , as I said, in most cases the problem is not a Common Lisp problem, and developers working in other languages run into the same issues.

The good news is that we know that these libraries worked on some C or C++ project, with the right SOs or DLLs, so it should most likely also run on Common Lisp, and that is what we have found so far. Which means that, with perseverance, we can get our game working, and learn how to smooth all the wrinkles, and that is part of the value we get from this, “Game 2” should be easier to develop once we know where the bumps on the road were when creating “Game 1”, and so forth.

A lot of the libraries we are using are a labor of love. As far as I know Baggers does not receive any remuneration for developing CEPL, and I think neither does Shinmera for working on Qtools, to mention just a couple, some projects are more complex than others, going from simple bindings, to completely lispified ways of writing code, so the parts are there, we have to piece them together, document our findings, and help to improve the existing tools however we can.

I think that making games, will improve the tools, which will make it easier to produce more games, which in turn will show us more ways of improving the tools, and that continuous spiral will make Lisp as good or better than other languages in terms of game development, and I think we have enough people writing (or trying to write) games to make this happen.

Releated Links

Reddit: u/maufdez
Twitter: @MauricioFdezF
Masstodon: @maufdez@fosstodon.org
Github: maufdez
Blog: Funlisp
Google+: MauricioFernandezF