Elixir Game Devlog 0x09 - Game Database - Importing all JSON files

Phew, It's nearly time I implemented a Game Database!
I mean, using JSON files for holding data is great but working with it in the format provided isn't.
Loading each file and converting it to JSON and then loading another one and then trying to combine them all makes it extremely difficult to code something worthwhile.

You might be thinking: Yeah, I'm going to use (no)SQL for the task! Nope, SQL is not up untill later in the development process.

I first need to import all 606 JSON data files. Convert eachone of them to a Elixir data structure. Then iterate over each data structure list and all it's objects.
Then add each one of them to a unified Map/Structure in memory database which  can be searched.

After the script is done done it can then save this database to a file. That file will be loaded into a process and accessed afterwards throughout the development process..

Later I might convert it to a SQL database or something similair so we gain development speed.
But frankly, the files are 10 MB. If we use around 10-20MB worth of RAM then it's really no need to complicate things with a database for the JSOn files.

Removed any non json file from the data/json folder (including some extracts from thes source code)
Moved (and modified) data/json/uncraft/classes/ammo.json as it misbihaved and crashed our server.

The current object handling implementation is pretty simple

Current object handling implementation

  def handle_object(object) doid =   object["id"]type = object["type"]abstract = object["abstract"]IO.inspect objectif !is_nil(id) or !is_nil(abstract) do\tif is_nil(id), do: id = abstract\tcreate_category_for_type(type, %{})\tDataStorage.add(@database,[type,id], object)else  create_category_for_type(type, [])#  IO.inspect object  DataStorage.append(@database,type,object)end  end

WOW. I didn't believe it. But almost 10 MB of json was translated to 1 file of 347.5 KB with 1 big elixir map.
I still have to look into it to view if everything is there and nothing is missing as I suspect something might have gone wrong:)

Yup, upon looking into it.. it seems that data is TRUNCATED when we try to save it as a map.
Seems that saving it with :erlang.term_to_binary works and the file size is now 7.9MB
It could have been larger but term_to_binary actually zips the data ☺

Upon further investigatin I found out that I can use a special setting to allow infinity data to be printed out.. Great.
I guess I'll save both term and map data for the time being.

IO.inspect(large_string, printable_limit: :infinity, limit: :infinity)

And it's now 8.7 MB .

By looking at the JSOn source I can see that the uncraft type is special and that it contains a result key which could take as sub ID for the uncraft key instead of using it as a list.
I've also noticed that it's interesting to select the ID and be able to show the user a name instead of the id. This will be useful when searching for names to show the user for nondescript ID's.

Let's just store the id and name reference in a different key I'll call "names"

For now I'm content with the current Database I've imported and saved to file. It could be better but I can now start prototyping other aspectsof the game.
You can view certain changes here https://lba.im/fossilr3po/lifebeyondapocalypse-elixir/info/135070ed01d6e6ec
Next we'll look at an improved and extended item searching functionality