Network, downloading and using URLs

1. Loading and saving files using URLs

All methods in our engine take URL as the parameter, not just a FileName. Although in most cases you can also pass a filename (absolute or relative to the current directory), and it will also work as expected.

All loading and saving routines (for models, images, sounds, and all other resources) automatically deal with URLs.

To directly load or save your own binary file formats (as ObjectPascal TStream):

• To load (easily), use a simple Download function. It simply returns a TStream that contains the resource indicated by the URL. It supports all the protocols mentioned below, e.g. file, castle-data.

  It can even download data using http or https protocols, although you need to set EnableBlockingDownloads to true for this.

• To load asynchronously (to continue the flow of your application while the download takes place in the background), use the TCastleDownload class. It presents a trivial API to start and watch the download progress, and offers a lot of features for HTTP requests. When the TCastleDownload.Status is dsSuccess you have the data (as a TStream) inside TCastleDownload.Contents.

  It supports all our procotols. It can download data using http or https protocols without any issues. It can be used to communicate with a REST server.

  Examples:

  • examples/network/asynchronous_download/ demonstrates multiple simultaneous downloads, along with a 3D animation, running smoothly.

  • examples/network/castle_download/ implements a simple command-line downloading tool (like wget or curl using our engine).

  • examples/network/remote_logging/ sends asynchronous HTTP POST message.

• To save, use URLSaveStream function. Right now, it can only save to a local file, so it merely translates a URL to local filename and creates a TFileStream for you. Still, it's a good idea to use it, to uniformly deal with URLs throughout your application.

If you want to read or write text files from an URL, use TTextReader and TTextWriter.

2. Supported protocols

2.1. Downloading from the network: http and https

http and https work. You can download data from the Internet, and the TCastleDownload has a support for various HTTP methods (GET, POST). You can use this for simple downloading, or for full-featured communication with a REST server.

Asynchronous TCastleDownload supports http and https automatically. It is perfect to use with unreliable / slow network.

Synchronous Download supports these protocols only if you set global variable EnableBlockingDownloads to true. We call them "blocking downloads" because the application simply waits for the un-interruptible download to finish. This is easy to use, but may cause your application to hang, as network may be slow / unreliable. We advise using TCastleDownload for network protocols, although it requires a bit more effort.

For the https (encrypted version of http) protocol to work:

1. You need to use FPC >= 3.2.0. Older FPC versions have critical problem with this.

2. Use the OpenSSLSockets unit. Simply add this to the uses clause of one of your units (like GameInitialize, if you follow our conventions for cross-platform games)):

   {$ifndef VER3_0} OpenSSLSockets, {$endif} // support HTTPS

3. You need to also distribute OpenSSL library. On Linux and FreeBSD, it is almost for sure installed on user's system already. On Windows, use the appropriate DLL.

   Our build tool (used also if you use our editor) takes care of it for you. Simply add <dependency name="Https" /> in your CastleEngineManifest.xml.

Internally, we use FpHttpClient unit, which supports http and https.

Note that (almost) all of the parameters and attributes in the engine API are URLs. So you can refer to network resources (http, https) anywhere, and it should "just work".

For example, your game level data may be actually downloaded from the network when loading level. To do this, level.xml may use an http protocol when referring to a scene. Like this:

<?xml version="1.0"?>
<level
  name="pits"
  type="Level"
  scene="https://raw.githubusercontent.com/castle-engine/castle-engine/master/tools/castle-editor/data/project_templates/3d_fps_game/files/data/level/level-dungeon.gltf"
  title="The Pits of Azeroth"
/>

and the scene with all associated resources will be downloaded.

Inside models (like X3D, glTF and other), you can also refer to network resources, and it will "just work". For example you can use X3D Inline node to inline a model from given URL, you can use X3D Anchor node to switch to given model on click, you can refer to textures and sounds and scripts and everything else from the network. Relative URLs are always resolved with respect to the containing document.

On Android, you should use the download_urls service to support http and https protocols.

2.2. Loading local files: file

To load normal files from disk, use a file URL.

Absolute file URLs look like this: file:///c:/windows/clock.avi (on Windows) or file:///etc/fstab (on Unix). You can also use normal absolute filenames with most CGE routines, like c:\windows\clock.avi (on Windows; you can use slash or backslash) or /etc/fstab (on Unix).

In most cases absolute filenames are not very useful (since they would be specific to a particular system). Using relative URLs makes more sense, like textures/wood.png. Relative URLs should use slashes, and work naturally when used in other files (relative URL is then relative to the containing file) or code (relative URL is then relative to the current working directory). Note that the current working directory depends on how the user runs your application.

To reliably load game data from code you should use castle-data protocol, not file protocol.

CastleURIUtils contains routines to operate on URLs (and more general URIs), including converting between regular filenames and URLs with file: protocol.

• Use this to convert a FileName (relative or absolute) to an absolute URL.

  URL := FilenameToURISafe(FileName);

• Use this to convert something that may be a FileName or URL to an URL. This is safer than FilenameToURISafe(...), in that it will never touch something that already is an URL.

  URL := AbsoluteURI(FileNameOrURL);

• Use this to convert URL back to a FileName. When the URL is a file: protocol, it will decode back the simple filename. Right now, URL without protocol is also returned back as a simple filename. When the URL uses a different protocol (like http), returns empty string.

  FileName := URIToFilenameSafe(URL);

See reference of FilenameToURISafe, AbsoluteURI, URIToFilenameSafe.

If you read/write filenames from/to Lazarus classes, for example if you use Lazarus TOpenDialog.FileName or TSaveDialog.FileName, you can use the UTF-8 variants instead: URIToFilenameSafeUTF8 and FilenameToURISafeUTF8. But it doesn't matter in practice. Both Castle Game Engine and Lazarus configure FPC RTL to use UTF-8 for all strings (by default FPC RTL uses system-dependent encoding).

On Android, you should use the read_external_storage service to be able to read storage files (e.g. from SD card) through the file protocol.

2.3. Loading data files: castle-data

This protocol should be used to load data files of your project. During development, on normal desktop systems (Windows, Linux etc.), the data files are simply files inside the data subdirectory of your project. You should place there all the files loaded at runtime by your application.

When the application is packaged for some systems, like Android or iOS, the data directory may be treated in a special way. If you access all the data files using the castle-data protocol (or using URLs relative to files loaded using the castle-data protocol) then your application will "just wok" on all systems.

See the documentation about the data directory.

Note that you can adjust ApplicationDataOverride to host your data files wherever you want. This way data files may even be loaded from http location. On desktop systems, the data location is by default just a regular directory on disk, but you can change it.

Loading from the castle-data:/images/my_image.png is equivalent to using ApplicationData in code and loading from the ApplicationData('images/my_image.png'). Since Castle Game Engine 6.5, we advise using castle-data:/images/my_image.png form.

2.4. Embedded data: data

data is a special protocol that doesn't refer to an external resource. Instead, the complete data URI contains the contents. This allows to embed various resources (like textures, sounds, other 3D models) inside a parent file. For example instead of referring to the texture filename from 3D model — you can embed the actual texture contents inside 3D model file. This is sometimes a very nice feature (it makes the file easier to distribute).

See data: URI specification. Our engine includes a tool to-data-uri that can turn any file into a data URI, and you can use such data URI everywhere where we expect URL. to-data-uri is provided in the regular engine download in the bin/ subdirectory, also source code is here.

Wherever our engine, or X3D, says that it expects a URL — you can use data URI to provide the contents "right there", without using any additional file.

Demos of using data URI are inside our VRML/X3D demo models, see in particular x3d/data_uri.x3dv.

2.5. (Internal) Android assets: castle-android-assets

This protocol is called castle-android-assets or (deprecated name) assets. It is only available on Android.

Used to access data files in an Android application. "Asset" files live inside your application's .apk file, together with your compiled game. The build tool will copy the data directory of your game to Android assets. For example, file that was in data/my_texture.png in your source code can be accessed (from the Android app) using the URL assets:/my_texture.png.

You should never explicitly use this protocol name, as it does not work on other platforms than Android. Instead, use ApplicationData to refer to your data files from code. The ApplicationData will always return an absolute URL to the data file location on current platform. On Android it will start with castle-android-assets:/... but you should treat this as an internal detail.

3. Dialog windows that support URLs

If you use TCastleWindow, it gives you a ready TCastleWindow.FileDialog that takes and returns URLs.

If you use Lazarus with TCastleControl, we advise to use our dialog components: TCastleOpenDialog, TCastleSaveDialog, TCastleOpen3DDialog, TCastleOpenImageDialog, TCastleSaveImageDialog. They expose URL property which works naturally with CGE.

4. Notes about terminology: URI vs URL

URI is a more general term. URI uniquely identifies a resource but does not necessarily tell us how to load (download) or save (upload) it. We have many routines in CastleURIUtils unit that process URIs (strings), they use the more general term URI. They complement standard FPC URIParser routines.

URL is a specific type of URI that also tells you how to load or save the resource. For example http and file protocols define URLs. Most of our routines that load or save use the term URL.

Things get a little more cloudy when you realize there's also data URI scheme. It's not precisely an URL (it's not an address of a resource), but you can load it (since the URI itself contains the resource). And we support it fully (our Download method loads it automatically). Admittedly, this means that our loading routines should rather use the term URL or data URI, but that's just long and (for those who don't use data URI) confusing, so for simplicity we just keep (over-)using the term URL. Also, other standards (like CSS and X3D and VRML) allow placing data URIs inside fields called url.

If you enjoy reading about Internet terminology, note that we use in our engine also URNs (another subtype of URI). They are used by X3D external prototypes, see X3D extensions introduction.

5. Multi-player options

5.1. Indy (CGE example using TCP streams)

The examples/network/tcp_connection directory in CGE sources demonstrates how to create and use a classic client/server solution, where multiple clients talk to a server over a TCP/IP connection.

It's a good cross-platform solution when:

• The client/server architecture fits your design. That is: one player "hosts" a game, and everyone can reach the host over IP (which typically means that either 1. the host IP, and the relevant port, are accessible publicly on the Internet, 2. or that everyone is within the same local network).

• You want reliability (not maximum speed), since it uses TCP connection in a standard fashion.

• CastleClientServer and Indy use standard TCP connection in a standard way, which is good for simplicity and interoperability. E.g. you could develop a server in Java or C++ if needed, to communicate with Pascal clients.

This approach uses the CastleClientServer unit, which uses Indy (with threads) on most platforms, except on Android where we utilize dedicated asynchronous Android API for this.

To compile it, be sure to install Indy using one of the options below, and then also compile in Lazarus package castle_indy. To install Indy:

• You can install Indy through the Online Package Manager. The OPM is a great way to install Lazarus packages in general, go ahead and try it :)

• You can install "indy" module using fpcupdeluxe. You can install your own FPC and Lazarus using fpcupdeluxe, and add an "indy" module to it.

• Alternatively, you can download Indy from packages.lazarus-ide.org (same packages that OPM uses) . Search for "indy" there, download and unpack the zip, open the package indylaz.lpk inside Lazarus and compile it. Here's a command-line version:

  wget http://packages.lazarus-ide.org/Indy10.zip
  unzip Indy10.zip
  lazbuild Indy10/indylaz.lpk
  

In all cases, you should get an additional package indylaz known by Lazarus. Remember to also install packages/castle_indy.lpk package, and use it in your projects.

5.2. RNL (CGE example of real-time online shooter)

RNL (Realtime Network Library) is an open-source, reliable UDP network library, for FPC and Delphi, cross-platform, developed by Benjamin Rosseaux . If you want to make real-time communication over a network (e.g. an FPS game like Quake) this is a good start.

In fact, we have already made it :) Not Quake is an example of online first-person shooter, developed using Castle Game Engine and RNL. You can

• Get source code and binary releases of client and server from GitHub

• Get binary releases of client from Itch.io

• Read the news post

5.3. Other options

There are various other networing solutions for Pascal — and you can use any of them together with Castle Game Engine. Links:

• Aforementioned Indy is a big library providing a lot of networking options. You can use it directly in many ways. See the online documentation.

• Synapse is a cross-platform networking library. See also FPC wiki about Synapse.

• lNet is a cross-platform lightweight networking library for FPC. It's much smaller (in terms of API and implementation) than Synapse and Indy, which may be an advantage, depending on what you need. See lNet FAQ and FPC wiki about lNET.

• FPC includes some networking units in the standard installation already. They work at various levels. In particular if you just want HTTP (REST) networking, FPC has fcl-web which allows to create HTTP(S) servers and clients.

• Remember about our Download function and TCastleDownload class. They use FpHttpClient under the hood. You can use TCastleDownload for HTTP(S) communiction with a REST server written in any language (Pascal or not).

The future may bring to Castle Game Engine more networking options (at a higher-level, to replicate some game state across multiple clients).

Mote that you can also make hot seat and split screen games, in which case multiple people just play on the same computer. We fully support multiple joysticks connected to a single desktop application, and connecting / disconnecting them at runtime, which allows to handle input from multiple people in one game.