URLs, loading (downloading) and saving resources

Table of Contents

1. Introduction
2. Loading and saving any URL using TStream, synchronously or asynchronously
3. Loading and saving specialized resources
4. Case-sensitive filesystems and URLs
5. Supported protocols
6. Dialog windows that support URLs
7. Notes about terminology: URI vs URL

1. Introduction

All methods in our engine take URL as the parameter, not just a filename. Although in most cases you can also pass a filename (absolute or relative to the current directory), and it will also work as expected.

All loading and saving routines (for models, images, sounds, and all other resources) automatically deal with URLs.

2. Loading and saving any URL using TStream, synchronously or asynchronously

To directly load or save your own files using ObjectPascal TStream:

To load (easily), use the Download function. It simply returns a TStream that contains the resource indicated by the URL. It supports all the protocols mentioned below, e.g. file, castle-data.

It can also load data from a ZIP file if you associate a custom protocol with your ZIP file. See TCastleZip.RegisterUrlProtocol.

It can even download data using http or https protocols, although you need to set EnableBlockingDownloads to true for this.
To load asynchronously (to continue the flow of your application while the download takes place in the background), use the TCastleDownload class. It presents a trivial API to start and watch the download progress, and offers a lot of features for HTTP requests. When the TCastleDownload.Status is dsSuccess you have the data (as a TStream) inside TCastleDownload.Contents.

It supports all our protocols. It can download data using http or https protocols without any issues.

It can be used to communicate with a REST server.

Examples:
- examples/network/asynchronous_download/ demonstrates multiple simultaneous downloads, along with a 3D animation, running smoothly.
- examples/network/castle_download/ implements a simple command-line downloading tool (like wget or curl using our engine).
- examples/network/remote_logging/ sends asynchronous HTTP POST message.
- castle-openai: Talking with OpenAI assistant (essentially your own, customized ChatGPT) is an example of using CastleDownload to communicate with a REST server.
To save, use UrlSaveStream function.

It can save to a local file.

It can also write to a ZIP file if you associate a custom protocol with your ZIP file. See TCastleZip.RegisterUrlProtocol.

3. Loading and saving specialized resources

More higher-level loading and saving routines are available. They are all implemented on top of the above functions dealing with TStream, so they all support all registered URLs.

To read / write text files from an URL, use TTextReader and TTextWriter.
To read / write text files as a simple String from an URL, use FileToString and StringToFile.
To read / write 3D models, use LoadNode and SaveNode.

Or (advised for most cases) use higher-level classes for 3D models: just set TCastleSceneCore.Url to read and call TCastleSceneCore.Save to write. See viewport and scenes for more information.
To read / write images, use LoadImage and SaveImage.

Or (advised for most cases) use even higher-level classes for displaying images: TCastleImageTransform, TCastleImageControl. See using images for more information.
To read audio files, use TCastleSound. See sound for more information.
To read / write XML files, use UrlReadXML and UrlWriteXML.
To read / write serialized components in JSON files, use routines in CastleComponentSerialize unit.

4. Case-sensitive filesystems and URLs

Often, URLs just refer to files on the filesystem. Like file:// URLs, or castle-data:/ URLs (on platforms where the application data is just a set of regular files). In this case, the underlying filesystem determines whether the names are case-sensitive (so e.g. foobar vs FooBar mean something else) or not.

On most operating systems (like Linux, FreeBSD, macOS; but also mobile and console filesystems) the filesystems are typically case-sensitive.
On Windows, the filesystems are typically not case-sensitive.

Note	To complicate matters, whether the filesystem is case sensitive or not is not actually determined by the OS. You can mount on Windows a case-sensitive filesystem like Ext4. You can mount on Linux a case-ignoring filesystem like NTFS. So the filesystems can be case-sensitive or not on any OS.

To make the application work on all platforms, be sure to always specify the same case in URLs as your files. So, assume that URLs are case-sensitive.

E.g. take care if you load castle-data:/FooBar.png or castle-data:/foobar.png. Using wrong letter case may be an easy mistake, because on Windows both versions will work, but on Linux only the version with correct case.

If you don’t want to care about the latter case (which makes sense if you develop mostly on Windows but want your application to also work on other platforms), a simple solution is to set global variable CastleDataIgnoreCase to true.

5. Supported protocols

5.1. Loading local files: `file`

To load normal files from disk, use a file URL.

Absolute file URLs look like this: file:///c:/windows/clock.avi (on Windows) or file:///etc/fstab (on Unix). You can also use normal absolute filenames with most engine routines, like c:\windows\clock.avi (on Windows; you can use slash or backslash) or /etc/fstab (on Unix).

In most cases hardcoding an absolute filename in your application is not very useful (since it would be specific to a particular system). Using relative URLs makes more sense, like textures/wood.png. Relative URLs should use slashes, and work naturally when used in other files (relative URL is then relative to the containing file) or code (relative URL is then relative to the current working directory). Note that the current working directory depends on how the user runs your application.

To reliably load game data from code you should use castle-data protocol, not file protocol.

CastleUriUtils contains routines to operate on URLs (and more general URIs), including converting between regular filenames and URLs with file: protocol.

Use this to convert a FileName (relative or absolute) to an absolute URL:
```
URL := FilenameToUriSafe(FileName);
```
Use this to convert something that may be a FileName or URL to an URL. This is safer than FilenameToUriSafe(…), in that it will never touch something that already is an URL.
```
URL := AbsoluteUri(FileNameOrUrl);
```
Use this to convert URL back to a FileName. When the URL is a file: protocol, it will decode back the simple filename. Right now, URL without protocol is also returned back as a simple filename. When the URL uses a different protocol (like http), returns empty string.
```
FileName := UriToFilenameSafe(URL);
```

See reference of FilenameToUriSafe, AbsoluteUri, UriToFilenameSafe.

On Android, you should use the read_external_storage service to be able to read storage files (e.g. from SD card) through the file protocol.

5.2. Loading data files: `castle-data`

This protocol should be used to load data files of your project. During development, on normal desktop systems (Windows, Linux etc.), the data files are simply files inside the data subdirectory of your project. You should place there all the files loaded at runtime by your application.

When the application is packaged for some systems, like Android or iOS, the data directory may be treated in a special way. If you access all the data files using the castle-data protocol (or using URLs relative to files loaded using the castle-data protocol) then your application will "just wok" on all systems.

See the documentation about the data directory.

Note that you can adjust ApplicationDataOverride to host your data files wherever you want. This way data files may even be loaded from http location. On desktop systems, the data location is by default just a regular directory on disk, but you can change it.

5.3. Writing data (specific to given user): `castle-config`

If you need to write some data, use the castle-config:/ protocol to access space for writing resources (config files, savegames, user projects…). For example

var
  MyUrl: String;
  MyStream: TStream;
begin
  MyUrl := 'castle-config:/my_data.txt';
  MyStream := UrlSaveStream(MyUrl);
  try
    // TODO: write to MyStream whatever you need
  finally
    FreeAndNil(MyStream);
  end;
end;

Notes:

The URLs castle-config:/xxx point to writeable resources.

If case of regular desktops, these are just files and opening them with UrlSaveStream does underneath just TFileStream.Create(FileName, fmCreate).
We follow OS-specific conventions and APIs to determine a space for writing user-specific data. For example castle-config:/my_file.txt maps to:
- On Linux and FreeBSD: /home/<username>/.config/<application-name>/my_file.txt
- On Windows: C:/Users/<username>/AppData/Local/<application-name>/my_file.txt
To be more precise, on most systems we use logic from FPC GetAppConfigDir function. (When using Delphi, we have a compatibility implementation with similar logic.) This in turn looks at ApplicationProperties.ApplicationName and follows the OS-specific conventions and APIs to determine the best config directory. On UNIX this follows XDG Base Directory Specification.
The contents saved to castle-config:/xxx are persistent across program runs. Use it for savegames, databases (e.g. using SQLite), user preferences, etc.

Our UserConfig is also saved there. It’s just in file castle-config:/<application-name>.conf.

Note

Using the <application-name> for UserConfig location is somewhat superfluous, as it is already stored in the castle-config:/ directory that is application-specific. But it does seem consistent with what some other applications are doing and makes it easy to find the file in weird situations, in case it gets mixed up with other files.

Using the castle-config:/ URL literally (without anything more) indicates the top-level directory of the user configuration. It is initially (when user first runs the application) empty. But you can create as many files there as you want.

If you want to initialize something in your config based on your data, do it explicitly — e.g. like this:

uses SysUtils, CastleClassUtils;

procedure MakeSureMyStuffExistsInConfig;
var
  OutputStream, InputStream: TStream;
begin
  if not UriExists('castle-config:/my_stuff.data') then
  begin
    OutputStream := UrlSaveStream('castle-config:/my_stuff.data');
    try
      InputStream := Download('castle-data:/my_stuff_initial.data');
      try
        ReadGrowingStream(InputStream, OutputStream, true);
      finally FreeAndNil(InputStream) end;
    finally FreeAndNil(OutputStream) end;
  end;
end;

The subdirectories inside castle-config:/ are automatically created as you save files using UrlSaveStream. So e.g. the code below "just works" and will create my_subdirectory automatically:
```
MyStream := UrlSaveStream('castle-config:/my_subdirectory/my_data.txt');
```

5.4. Downloading from the network: `http` and `https`

http and https work. You can download data from the Internet, and the TCastleDownload has a support for various HTTP methods (GET, POST). You can use this for simple downloading, or for full-featured communication with a REST server.

Asynchronous TCastleDownload supports http and https automatically. It is perfect to use with unreliable / slow network.

Synchronous Download supports these protocols only if you set global variable EnableBlockingDownloads to true. We call them "blocking downloads" because the application has to simply wait for the download to finish and there’s no way to interrupt the download (without just killing the application) or even watch the progress.

We advise always using TCastleDownload to get data from the network — although it requires a bit more effort from code, but allows to observe and interrupt the download.

For the https (encrypted version of http) protocol to work:

For FPC: Use the OpenSSLSockets unit. Simply add this to the uses clause of one of your units (like GameInitialize):
```
{$ifdef FPC} OpenSSLSockets, {$endif} // support HTTPS
```
Make sure users have the OpenSSL library installed.

On Unix (Linux, FreeBSD, macOS…), it is standard to have OpenSSL installed already system-wide. Developers and users likely don’t need to do anything.

On Windows, use the appropriate DLLs. Our editor (or command-line build tool) will automatically place the proper DLLs alongside your EXE file on the first build. You only need to add <dependency name="Https" /> in your CastleEngineManifest.xml.

See an example like examples/network/asynchronous_download/ that does both things above, to make https work.

Note that we use URLs, not filenames, throughout the entire engine API. So to load something from the network, you can just pass e.g. https://…; to TCastleSceneCore.Load.

Inside models (like X3D, glTF and other), you can also refer to network resources, and it will "just work". For example you can use X3D Inline node to inline a model from given URL, you can use X3D Anchor node to switch to given model on click, you can refer to textures and sounds and scripts and everything else from the network. Relative URLs are always resolved with respect to the containing document.

On Android, you should use the download_urls service to support http and https protocols.

5.5. Embedded data: `data`

data is a special protocol that doesn’t refer to an external resource. Instead, the complete data URI contains the contents. This allows to embed various resources (like textures, sounds, other 3D models) inside a parent file. For example instead of referring to the texture filename from 3D model — you can embed the actual texture contents inside 3D model file. This is sometimes a very nice feature (it makes the file easier to distribute).

See data: URI specification. Our engine includes a tool to-data-uri that can turn any file into a data URI, and you can use such data URI everywhere where we expect URL. to-data-uri is provided in the regular engine download in the bin/ subdirectory, also source code is here.

Wherever our engine, or X3D, says that it expects a URL — you can use data URI to provide the contents "right there", without using any additional file.

Demos of using data URI are inside our demo models, see in particular x3d/data_uri.x3dv.

5.6. Data from a zip

You can open and register an arbitrary ZIP file to read / write resources from it using a custom URL. See the TCastleZip.RegisterUrlProtocol docs.

See the examples/network/custom_url_handler, when USE_ZIP_URL_HANDLER is defined.

5.7. Registering your own protocol

You are not limited to the protocols documented here. You can register your own URL protocols, with custom reading and writing handlers, using RegisterUrlProtocol.

See the examples/network/custom_url_handler, when USE_ZIP_URL_HANDLER is undefined.

6. Dialog windows that support URLs

If you use TCastleWindow, it gives you a ready TCastleWindow.FileDialog that takes and returns URLs.

If you use Lazarus with TCastleControl, we advise to use our dialog components: TCastleOpenDialog, TCastleSaveDialog, TCastleOpenSceneDialog, TCastleOpenImageDialog, TCastleSaveImageDialog. They expose URL property which works naturally with CGE.

7. Notes about terminology: URI vs URL

URI is a more general term. URI uniquely identifies a resource but does not necessarily tell us how to load (download) or save (upload) it. We have many routines in CastleUriUtils unit that process URIs (strings), they use the more general term URI. They complement standard FPC URIParser routines.

URL is a specific type of URI that also tells you how to load or save the resource. For example http and file protocols define URLs. Most of our routines that load or save use the term URL.

Things get a little more cloudy when you realize there’s also data URI scheme. It’s not precisely an URL (it’s not an address of a resource), but you can load it (since the URI itself contains the resource). And we support it fully (our Download method loads it automatically). Admittedly, this means that our loading routines should rather use the term URL or data URI, but that’s just long and (for those who don’t use data URI) confusing, so for simplicity we just keep (over-)using the term URL. Also, other standards (like CSS and X3D) allow placing data URIs inside fields called url.

If you enjoy reading about Internet terminology, note that we use in our engine also URNs (another subtype of URI). They are used by X3D external prototypes.

To improve this documentation just edit this page and create a pull request to cge-www repository.