Network, downloading and using URLs

1. Loading and saving files using URLs

All methods in our engine take URL as the parameter, not just a FileName. Although in most cases you can also pass a filename (absolute or relative to the current directory), and it will also work as expected.

All loading and saving routines (for models, images, sounds, and all other resources) automatically deal with URLs.

To directly load or save your own binary file formats (as ObjectPascal TStream):

• To load (easily), use a simple Download function. It simply returns a TStream that contains the resource indicated by the URL. It supports all the protocols mentioned below, e.g. file, castle-data.

  It can even download data using http or https protocols, although you need to set EnableBlockingDownloads to true for this.

• To load asynchronously (to continue the flow of your application while the download takes place in the background), use the TCastleDownload class. It presents a trivial API to start and watch the download progress, and offers a lot of features for HTTP requests. When the TCastleDownload.Status is dsSuccess you have the data (as a TStream) inside TCastleDownload.Contents.

  It supports all our protocols. It can download data using http or https protocols without any issues. It can be used to communicate with a REST server.

  Examples:

  • examples/network/asynchronous_download/ demonstrates multiple simultaneous downloads, along with a 3D animation, running smoothly.

  • examples/network/castle_download/ implements a simple command-line downloading tool (like wget or curl using our engine).

  • examples/network/remote_logging/ sends asynchronous HTTP POST message.

  • castle-openai: Talking with OpenAI assistant (essentially your own, customized ChatGPT) is an example of using TCastleDownload to communicate with a REST server.

• To save, use UrlSaveStream function. Right now, it can only save to a local file, so it merely translates a URL to local filename and creates a TFileStream for you. Still, it's a good idea to use it, to uniformly deal with URLs throughout your application.

If you want to read or write text files from an URL, use TTextReader and TTextWriter.

2. Note about case-sensitive filesystems and URLs

Often, URLs just refer to files on the filesystem. Like file://... URLs, or castle-data:/... URLs (on platforms where the application data is just a set of regular files). In this case, the underlying filesystem determines whether the names are case-sensitive (so e.g. foobar vs FooBar mean something else) or not. On Unix (like Linux, FreeBSD, macOS) the filesystems are typically case-sensitive. On Windows, the filesystems are typically not case-sensitive.

To make the application work on all platforms, be sure to always specify the same case in URLs as your files.

E.g. take care if you load castle-data:/FooBar.png or castle-data:/foobar.png. Using wrong letter case may be an easy mistake, because on Windows both versions will work, but on Linux only the version with correct case.

If you test and develop mostly on Windows, a simple option is also be to set global CastleDataIgnoreCase:=true. See CastleDataIgnoreCase.

3. Supported protocols

3.1. Downloading from the network: http and https

http and https work. You can download data from the Internet, and the TCastleDownload has a support for various HTTP methods (GET, POST). You can use this for simple downloading, or for full-featured communication with a REST server.

Asynchronous TCastleDownload supports http and https automatically. It is perfect to use with unreliable / slow network.

Synchronous Download supports these protocols only if you set global variable EnableBlockingDownloads to true. We call them "blocking downloads" because the application has to simply wait for the download to finish and there's no way to interrupt the download (without just killing the application) or even watch the progress. We advise always using TCastleDownload to get data from the network — although it requires a bit more effort from code, but allows to observe and interrupt the download.

For the https (encrypted version of http) protocol to work:

1. For FPC: Use the OpenSSLSockets unit. Simply add this to the uses clause of one of your units (like GameInitialize):

   {$ifdef FPC} OpenSSLSockets, {$endif} // support HTTPS

2. Make sure users have the OpenSSL library installed.

   On Unix (Linux, FreeBSD, macOS...), it is standard to have OpenSSL installed already system-wide. Developers and users likely don't need to do anything.

   On Windows, use the appropriate DLLs. Our editor (or command-line build tool) will automatically place the proper DLLs alongside your EXE file on the first build. You only need to add <dependency name="Https" /> in your CastleEngineManifest.xml.

See an example like examples/network/asynchronous_download/ that does both things above, to make https work.

Note that we use URLs, not filenames, throughout the entire engine API. So to load something from the network, you can just pass e.g. https://... to TCastleSceneCore.Load.

Inside models (like X3D, glTF and other), you can also refer to network resources, and it will "just work". For example you can use X3D Inline node to inline a model from given URL, you can use X3D Anchor node to switch to given model on click, you can refer to textures and sounds and scripts and everything else from the network. Relative URLs are always resolved with respect to the containing document.

On Android, you should use the download_urls service to support http and https protocols.

3.2. Loading local files: file

To load normal files from disk, use a file URL.

Absolute file URLs look like this: file:///c:/windows/clock.avi (on Windows) or file:///etc/fstab (on Unix). You can also use normal absolute filenames with most CGE routines, like c:\windows\clock.avi (on Windows; you can use slash or backslash) or /etc/fstab (on Unix).

In most cases absolute filenames are not very useful (since they would be specific to a particular system). Using relative URLs makes more sense, like textures/wood.png. Relative URLs should use slashes, and work naturally when used in other files (relative URL is then relative to the containing file) or code (relative URL is then relative to the current working directory). Note that the current working directory depends on how the user runs your application.

To reliably load game data from code you should use castle-data protocol, not file protocol.

CastleUriUtils contains routines to operate on URLs (and more general URIs), including converting between regular filenames and URLs with file: protocol.

• Use this to convert a FileName (relative or absolute) to an absolute URL.

  URL := FilenameToUriSafe(FileName);

• Use this to convert something that may be a FileName or URL to an URL. This is safer than FilenameToUriSafe(...), in that it will never touch something that already is an URL.

  URL := AbsoluteUri(FileNameOrUrl);

• Use this to convert URL back to a FileName. When the URL is a file: protocol, it will decode back the simple filename. Right now, URL without protocol is also returned back as a simple filename. When the URL uses a different protocol (like http), returns empty string.

  FileName := UriToFilenameSafe(URL);

See reference of FilenameToUriSafe, AbsoluteUri, UriToFilenameSafe.

On Android, you should use the read_external_storage service to be able to read storage files (e.g. from SD card) through the file protocol.

3.3. Loading data files: castle-data

This protocol should be used to load data files of your project. During development, on normal desktop systems (Windows, Linux etc.), the data files are simply files inside the data subdirectory of your project. You should place there all the files loaded at runtime by your application.

When the application is packaged for some systems, like Android or iOS, the data directory may be treated in a special way. If you access all the data files using the castle-data protocol (or using URLs relative to files loaded using the castle-data protocol) then your application will "just wok" on all systems.

See the documentation about the data directory.

Note that you can adjust ApplicationDataOverride to host your data files wherever you want. This way data files may even be loaded from http location. On desktop systems, the data location is by default just a regular directory on disk, but you can change it.

3.4. Embedded data: data

data is a special protocol that doesn't refer to an external resource. Instead, the complete data URI contains the contents. This allows to embed various resources (like textures, sounds, other 3D models) inside a parent file. For example instead of referring to the texture filename from 3D model — you can embed the actual texture contents inside 3D model file. This is sometimes a very nice feature (it makes the file easier to distribute).

See data: URI specification. Our engine includes a tool to-data-uri that can turn any file into a data URI, and you can use such data URI everywhere where we expect URL. to-data-uri is provided in the regular engine download in the bin/ subdirectory, also source code is here.

Wherever our engine, or X3D, says that it expects a URL — you can use data URI to provide the contents "right there", without using any additional file.

Demos of using data URI are inside our demo models, see in particular x3d/data_uri.x3dv.

3.5. (Internal) Android assets: castle-android-assets

This protocol is called castle-android-assets or (deprecated name) assets. It is only available on Android.

Used to access data files in an Android application. "Asset" files live inside your application's .apk file, together with your compiled game. The build tool will copy the data directory of your game to Android assets. For example, file that was in data/my_texture.png in your source code can be accessed (from the Android app) using the URL assets:/my_texture.png.

You should never explicitly use this protocol name, as it does not work on other platforms than Android. Instead, use ApplicationData to refer to your data files from code. The ApplicationData will always return an absolute URL to the data file location on current platform. On Android it will start with castle-android-assets:/... but you should treat this as an internal detail.

4. Dialog windows that support URLs

If you use TCastleWindow, it gives you a ready TCastleWindow.FileDialog that takes and returns URLs.

If you use Lazarus with TCastleControl, we advise to use our dialog components: TCastleOpenDialog, TCastleSaveDialog, TCastleOpenSceneDialog, TCastleOpenImageDialog, TCastleSaveImageDialog. They expose URL property which works naturally with CGE.

5. Notes about terminology: URI vs URL

URI is a more general term. URI uniquely identifies a resource but does not necessarily tell us how to load (download) or save (upload) it. We have many routines in CastleUriUtils unit that process URIs (strings), they use the more general term URI. They complement standard FPC URIParser routines.

URL is a specific type of URI that also tells you how to load or save the resource. For example http and file protocols define URLs. Most of our routines that load or save use the term URL.

Things get a little more cloudy when you realize there's also data URI scheme. It's not precisely an URL (it's not an address of a resource), but you can load it (since the URI itself contains the resource). And we support it fully (our Download method loads it automatically). Admittedly, this means that our loading routines should rather use the term URL or data URI, but that's just long and (for those who don't use data URI) confusing, so for simplicity we just keep (over-)using the term URL. Also, other standards (like CSS and X3D) allow placing data URIs inside fields called url.

If you enjoy reading about Internet terminology, note that we use in our engine also URNs (another subtype of URI). They are used by X3D external prototypes, see X3D extensions introduction.

6. Multi-player options

6.1. Use TCastleDownload for HTTP REST communication with a backend

The engine provides a cross-platform component TCastleDownload to asynchronously download any URL and to perform HTTP(S) web requests. In effect, this can be used to communicate with any backend (written using Pascal or not) that uses HTTP(S) protocol.

Features of TCastleDownload:

• Supports various HTTP(S) methods (GET, POST, PUT...).

• Handles HTTP(S) redirects automatically.

• Allows to send custom HTTP headers.

• Gets MIME type from server automatically.

• Exposes HTTP response headers and code.

• It is really cross-platform and cross-compiler, covering all platforms supported by Castle Game Engine. E.g. it uses FpHttpClient with FPC on most desktops, uses special Android service on Android, on Delphi uses Indy or TNetHTTPClient (depending on what works better on given platform).

• Supports encrypted HTTPS out-of-the-box as much as possible, since HTTPS is standard nowadays. To this end, we adjust some Indy and FpHttpClient to make HTTPS just work.

  FPC applications only have to use OpenSSLSockets unit, e.g. add

  {$ifdef FPC} OpenSSLSockets, {$endif}

  to the uses clause of one of your units (like GameInitialize).

Examples:

• examples/network/remote_logging/ (sends asynchronous HTTP POST message),

• examples/network/put_data/ (send HTTP PUT),

• castle-openai: Talking with OpenAI assistant (essentially your own, customized ChatGPT) (sends a series of HTTP POST and GET messages to talk with OpenAI server).

6.2. Indy (CGE example using TCP streams)

The examples/network/tcp_connection directory in CGE sources demonstrates how to create and use a classic client/server solution, where multiple clients talk to a server over a TCP/IP connection.

It's a good cross-platform solution when:

• The client/server architecture fits your design. That is: one player "hosts" a game, and everyone can reach the host over IP (which typically means that either 1. the host IP, and the relevant port, are accessible publicly on the Internet, 2. or that everyone is within the same local network).

• You want reliability (not maximum speed), since it uses TCP connection in a standard fashion.

• CastleClientServer and Indy use standard TCP connection in a standard way, which is good for simplicity and interoperability. E.g. you could develop a server in Java or C++ if needed, to communicate with Pascal clients.

This approach uses the CastleClientServer unit, which uses Indy (with threads) on most platforms, except on Android where we utilize dedicated asynchronous Android API for this.

To compile it with FPC/Lazarus, be sure to install Indy using one of the options below, and then also compile in Lazarus package castle_indy. To install Indy:

• You can install Indy through the Online Package Manager. The OPM is a great way to install Lazarus packages in general, go ahead and try it :)

• You can install "indy" module using fpcupdeluxe. You can install your own FPC and Lazarus using fpcupdeluxe, and add an "indy" module to it.

• Alternatively, you can download Indy from packages.lazarus-ide.org (same packages that OPM uses) . Search for "indy" there, download and unpack the zip, open the package indylaz.lpk inside Lazarus and compile it. Here's a command-line version:

  wget http://packages.lazarus-ide.org/Indy10.zip
  unzip Indy10.zip
  lazbuild Indy10/indylaz.lpk
  

In all cases, you should get an additional package indylaz known by Lazarus. Remember to also install packages/castle_indy.lpk package, and use it in your projects.

Delphi users don't need to do anything in this regard, as Indy is already included in Delphi.

6.2.1. Known memory leaks with Indy

All applications using Indy (whether with Castle Game Engine or not) have memory leaks, by default. This is a design choice of Indy — letting the memory leak was better than crashing in some edge-cases at unit finalization.

If you are sure that all the Indy threads are given sufficient time to terminate gracefully before the application exits, you can define the symbol FREE_ON_FINAL in Indy sources. Search for define FREE_ON_FINAL inside Indy sources, as of now it means you will find 5 include files doing this:

{.$DEFINE FREE_ON_FINAL}
{$UNDEF FREE_ON_FINAL}

You need to change these lines to

{$DEFINE FREE_ON_FINAL}
{.$UNDEF FREE_ON_FINAL}

Please note that it isn't always easy to guarantee that "threads are given sufficient time to terminate gracefully before the application exits". It means that disconnecting (by clients) and stopping (by server) cannot be done right when the application exits (or you have to wait for it to finish, potentially hanging your application on exit). Basically, Indy developers had a good reason to not enable this by default.

For similar reason, the internal thread inside CastleClientServer actually may leak memory in case it didn't have time to terminate gracefully before the application exits. (But in our case, by default, it will not leak if exits cleanly, i.e. you disconnect client before application exit. It only leaks if the application exits too quickly, disconnecting the client.)

Note: There's a code in Indy to "register known leak" with some memory managers, but it is not active by default for FPC.

You can test for memory leaks following our instructions about memory leaks. We use FPC HeapTrc to detect memory leaks, and you can just set <compiler_options detect_memory_leaks="true"> in CastleEngineManifest.xml to enable it.

6.2.2. Code completion using LSP (e.g. in VS Code) when using Indy

When you edit a project that uses CastleClientServer with VS Code (or other editor using our LSP) you may see errors like: "unit not found: IdGlobal".

To solve this, you need to tell our LSP where are your Indy units. To do this, create a file:

• On Unix: $HOME/.config/pasls/castle-pasls.ini

• On Windows: C:/Users/<YOUR-USERNAME>/AppData/Local/pasls/castle-pasls.ini

and inside put these lines:

[extra_options]
;; Specify as many extra FPC options as you want.
;; Each extra option must have a consecutive number, start from 1.
option_1=-Fu/home/myusername/src/Indy10/Core/
option_2=-Fu/home/myusername/src/Indy10/System/

Adjust the above paths (/home/myusername/src/Indy10) to your system to point to the directory where you have Indy sources. For more information, see README about Extra configuration in castle-engine/pascal-language-server.

6.3. RNL (CGE example of real-time online shooter)

RNL (Realtime Network Library) is an open-source, reliable UDP network library, for FPC and Delphi, cross-platform, developed by Benjamin Rosseaux . If you want to make real-time communication over a network (e.g. an FPS game like Quake) this is a good start.

In fact, we have already made it :) Not Quake is an example of online first-person shooter, developed using Castle Game Engine and RNL. You can

• Get source code and binary releases of client and server from GitHub

• Get binary releases of client from Itch.io

• Read the news post

6.4. Planned: Nakama integration

We plan integration with Nakama in the future. It's a great open-source solution that provides out-of-the-box common multi-player features and can be customized to each particular project.

6.5. Other options

There are various other networking solutions for Pascal — and you can use any of them together with Castle Game Engine. Links:

• Aforementioned Indy is a big library providing a lot of networking options. You can use it directly in many ways. See the online documentation.

• Synapse is a cross-platform networking library. See also FPC wiki about Synapse.

• lNet is a cross-platform lightweight networking library for FPC. It's much smaller (in terms of API and implementation) than Synapse and Indy, which may be an advantage, depending on what you need. See lNet FAQ and FPC wiki about lNET.

• FPC includes some networking units in the standard installation already. They work at various levels. In particular if you just want HTTP (REST) networking, FPC has fcl-web which allows to create HTTP(S) servers and clients.

The future may bring to Castle Game Engine more networking options (at a higher-level, to replicate some game state across multiple clients).

Mote that you can also make hot seat and split screen games, in which case multiple people just play on the same computer. We fully support multiple joysticks connected to a single desktop application, and connecting / disconnecting them at runtime, which allows to handle input from multiple people in one game.