Network, downloading and using URLs

1. Loading and saving files using URLs

All methods in our engine take URL as the parameter, not just a FileName. Although in most cases you can also pass a filename (absolute or relative to the current directory), and it will also work as expected.

All loading and saving routines (for 3D models, images, sounds, and all other resources) automatically deal with URLs. To actually load network URLs (like http or https) you only need to set CastleDownload.EnableNetwork to true.

To directly load or save your own binary file formats (as ObjectPascal TStream):

  • To load, use a simple Download function. It automatically handles all the details for you, including downloading from network (if EnableNetwork), and returns you a TStream that contains the resource indicated by the URL.

    An example examples/simple_command_line_utilities/castle_download.lpr uses this to implement a simple command-line downloading tool (like wget) using the engine.

  • To save, use URLSaveStream function. Right now, it can only save to a local file, so it merely translates a URL to local filename and creates a TFileStream for you. Still, it's a good idea to use it, to uniformly deal with URLs throughout your application.

If you want to read or write text files from an URL, use TTextReader and TTextWriter.

2. Supported protocols

2.1. Downloading from the network: http and https

Our engine can automatically download data from the network. All you have to do is set global EnableNetwork variable (from CastleDownload unit) to true. By default it is false, because for now the downloads are not user-friendly — they are blocking (we wait for them to finish, there's no way to cancel a download). This will be improved in the future, and eventually downloading from network may be enabled by default.

Internally, we use FpHttpClient unit, which supports http and (since FPC 3.0.2) https.

In order for the https to work, make sure that OpenSSL library is available. On Windows, you will probably want to place the appropriate DLLs alongside your exe file. You can find these DLLs inside the engine tools/build-tool/data/external_libraries/ subdirectory. These DLLs are also automatically included when packaging your application using the build tool, if you include <dependency name="Https" /> in your CastleEngineManifest.xml.

Note that (almost) all of the parameters and attributes in the engine API are URLs. So you can refer to network resources (http, https) anywhere, and it should "just work".

For example, your game level data may be actually downloaded from the network when loading level. To do this, level.xml may use an http protocol when referring to a scene. Like this:

<?xml version="1.0"?>
  title="The Pits of Azeroth"

and the scene with all associated resources will be downloaded.

Inside 3D models (like X3D, VRML and others), you can use network resources, for example you can Inline them (add a 3D model from another file), you can make Anchor to them, you can refer to textures and sounds and scripts and everything else from the network. Relative URLs are always resolved with respect to the containing document.

2.2. Loading local files: file

To load simple files from disk, just use a file URL. Of course this works regardless of the EnableNetwork value. These are just local filenames encoded as URL. CastleURIUtils contains routines to operate on URLs (and more general URIs), including converting between regular filenames and URLs with file: protocol.

  • Use this to convert a FileName (relative or absolute) to an absolute URL.

    URL := FilenameToURISafe(FileName);
  • Use this to convert something that may be a FileName or URL to an URL. This is safer than FilenameToURISafe(...), in that it will never touch something that already is an URL. On the other hand, there are some obscure cases (when a relative filename starts with a component with colon inside) when it may think that it has URL, while in fact it has a filename that should be converted to file: URL.
    URL := AbsoluteURI(FileNameOrURL);
  • Use this to convert URL back to a FileName. When the URL is a file: protocol, it will decode back the simple filename. Right now, URL without protocol is also returned back as a simple filename. When the URL uses a different protocol (like http), returns empty string.
    FileName := URIToFilenameSafe(URL);

See reference of FilenameToURISafe, AbsoluteURI, URIToFilenameSafe. See >doc/miscellaneous_notes/uri_filename.txt for more internal comments.

If you read/write filenames from/to Lazarus classes, for example if you use Lazarus TOpenDialog.FileName or TSaveDialog.FileName, use the UTF-8 variants instead: URIToFilenameSafeUTF8 and FilenameToURISafeUTF8. That is because Lazarus uses UTF-8 for all strings (as opposed to FPC RTL that uses system encoding).

2.3. Loading data files: castle-data

(Since Castle Game Engine >= 6.5).

This protocol should be used to load data files of your project. During development, on normal desktop systems (Windows, Linux etc.), the data files are simply files inside the data subdirectory of your project. You should place there all the files loaded at runtime by your application.

When the application is packaged for some systems, like Android or iOS, the data directory may be treated in a special way. If you access all the data files using the castle-data protocol (or using URLs relative to files loaded using the castle-data protocol) then your application will "just wok" on all systems.

See the documentation about the data directory.

Note that you can adjust ApplicationDataOverride to host your data files wherever you want. This way data files may even be loaded from http location. On desktop systems, the data location is by default just a regular directory on disk, but you can change it.

Loading from the castle-data:/images/my_image.png is equivalent to using ApplicationData in code and loading from the ApplicationData('images/my_image.png'). Since Castle Game Engine 6.5, we advise using castle-data:/images/my_image.png form.

2.4. Embedded data: data

data is a special protocol that doesn't refer to an external resource. Instead, the complete data URI contains the contents. This allows to embed various resources (like textures, sounds, other 3D models) inside a parent file. For example instead of referring to the texture filename from 3D model — you can embed the actual texture contents inside 3D model file. This is sometimes a very nice feature (it makes the file easier to distribute).

See data: URI specification. Our engine includes a tool examples/tools/to_data_uri.lpr that can turn any file into a data URI, and you can use such data URI everywhere where we expect URL.

Wherever our engine, or X3D, says that it expects a URL — you can use data URI to provide the contents "right there", without using any additional file.

Demos of using data URI are inside our VRML/X3D demo models, see in particular x3d/data_uri.x3dv.

2.5. (Internal) Android assets: castle-android-assets

This protocol is called castle-android-assets or (deprecated name) assets. It is only available on Android.

Used to access data files in an Android application. "Asset" files live inside your application's .apk file, together with your compiled game. The build tool will copy the data directory of your game to Android assets. For example, file that was in data/my_texture.png in your source code can be accessed (from the Android app) using the URL assets:/my_texture.png.

You should never explicitly use this protocol name, as it does not work on other platforms than Android. Instead, use ApplicationData to refer to your data files from code. The ApplicationData will always return an absolute URL to the data file location on current platform. On Android it will start with castle-android-assets:/... but you should treat this as an internal detail.

3. Dialog windows that support URLs

If you use TCastleWindowBase, it gives you a ready TCastleWindowBase.FileDialog that takes and returns URLs.

If you use Lazarus with TCastleControlBase, we advise to use our dialog components: TCastleOpenDialog, TCastleSaveDialog, TCastleOpen3DDialog, TCastleOpenImageDialog, TCastleSaveImageDialog. You can also continue using standard Lazarus dialog components. Our routines (almost) always handle a filename instead of an URL, or you can explicitly convert between filenames and URLs using functions mentioned earlier.

4. Notes about terminology: URI vs URL

URI is a more general term. URI uniquely identifies a resource but does not necessarily tell us how to load (download) or save (upload) it. We have many routines in CastleURIUtils unit that process URIs (strings), they use the more general term URI. They complement standard FPC URIParser routines.

URL is a specific type of URI that also tells you how to load or save the resource. For example http and file protocols define URLs. Most of our routines that load or save use the term URL.

Things get a little more cloudy when you realize there's also data URI scheme. It's not precisely an URL (it's not an address of a resource), but you can load it (since the URI itself contains the resource). And we support it fully (our Download method loads it automatically). Admittedly, this means that our loading routines should rather use the term URL or data URI, but that's just long and (for those who don't use data URI) confusing, so for simplicity we just keep (over-)using the term URL. Also, other standards (like CSS and X3D and VRML) allow placing data URIs inside fields called url.

If you enjoy reading about Internet terminology, note that we use in our engine also URNs (another subtype of URI). They are used by X3D external prototypes, see X3D extensions introduction.

5. Multi-player options

The examples/network/tcp_connection directory in CGE sources demonstrates how to create and use a "classic" client/server solution, where multiple clients talk to a server over a TCP/IP connection. The examples use CastleClientServer unit, which uses Indy (with threads) on most platforms, except on Android where we utilize dedicated asynchronous Android API for this.

It's a good cross-platform solution when:

  • The client/server architecture fits your design. That is: one player "hosts" a game, and everyone can reach the host over IP (which typically means that either 1. the host IP, and the relevant port, are accessible publicly on the Internet, 2. or that everyone is within the same local network).

  • You want reliability (not maximum speed), since it uses TCP connection in a standard fashion.

  • CastleClientServer and Indy use standard TCP connection in a standard way, which is good for simplicity and interoperability. E.g. you could develop a server in Java or C++ if needed, to communicate with Pascal clients.

There are various other networing solutions for Pascal — and you can use any of them together with Castle Game Engine. Links:

  • RNL (Realtime Network Library) by Benjamin Rosseaux is an open-source, reliable UDP network library, for both Delphi and FPC, cross-platform. If you want to make real-time communication (e.g. an FPS game like Quake) this may be a good start.

  • Aforementioned Indy is a big library providing a lot of networking options. See the online documentation.

  • Synapse is a cross-platform networking library. See also FPC wiki about Synapse.

  • lNet is a cross-platform lightweight networking library for FPC. It's much smaller (in terms of API and implementation) than Synapse and Indy, which may be an advantage, depending on what you need. See lNet FAQ and FPC wiki about lNET.

  • FPC includes some networking units in the standard installation already. They work at various levels. In particular if you just want HTTP (REST) networking, FPC has fcl-web which allows to create HTTP(S) servers and clients. Our Download function uses FpHttpClient under the hood as well — it's a simple and reliable HTTP(S) client.

The future may bring to Castle Game Engine more networking options (at a higher-level, to replicate some game state across multiple clients).

Mote that you can also make hot seat and split screen games, in which case multiple people just play on the same computer. We fully support multiple joysticks connected to a single desktop application, and connecting / disconnecting them at runtime, which allows to handle input from multiple people in one game.