Documentation upgrades done lately

Lots of documentation (in particular from wiki) was reworked, simplified, updated, consolidated with other pages… See in particular:

• Build Tool
• CastleEngineManifest.xml – Syntax and Examples
• Android Services (each particular service now has a dedicated documentation page)
• iOS Services (each particular service now has a dedicated documentation page)
• glTF
• Sprite sheets
• Coding conventions
• Lots of new pages are now linked from documentation sidebar. Browse them! 🙂

We converted 100% of our GitHub wiki contents to AsciiDoctor. This was done using kramdown-asciidoc and lots of manual tweaking.

Resulting AsciiDoctor sources are here. Editing them is trivial, and the HTMLs are auto-regenerated when updating the website. Note also the text To improve this documentation just edit the source of this page in AsciiDoctor (simple wiki-like syntax)…. at the bottom of new pages, with ready links to edit each page!

We have pretty URLs for pages generated by AsciiDoctor. So now we have https://castle-engine.io/build_tool instead of previous https://github.com/castle-engine/castle-engine/wiki/Build-Tool.

We also have nicer “404 not found” page now, try it: https://castle-engine.io/not_existing.

Why?

I was thinking lately of updates to our documentation. They come down to 2 big things:

1. Main page rework, to emphasize all our features using screenshots.

   For this, it is obvious I should just follow one of the numerous templates. We already use WordPress (for news) and own theme using Bootstrap. So we can easily adapt one of the numerous WordPress / bare Bootstrap templates.

2. Lots of manual updates, to document the fact that you can (and should, in simple cases) do many of the basic things using the editor. E.g. current manual describes usage of TCastleScene from code (and disregarding typical cross-platform app organization, that we advise). It should instead describe using TCastleScene from editor, and from code too, and placing most stuff in TUIState instances.

   I can pinpoint some pain-points of editing current docs in our PHP website:

   1. Writing API references is tiresome (need to spell out part of the generated URL).

   2. Adding images is tiresome (it breaks the workflow, to add new image, which is troublesome as lots of manual pages should get more screenshots).

   3. Formatting is a bit tiresome, and in effect we had docs spread between manual PHP/HTML and GitHub wiki.

Thoughts: Various approaches to documentation

I used a number of approaches for online documentation throughout my life, and within CGE.

• DocBook (e.g. for compositing shaders (to both HTML and PDF), my Ph.D thesis),

• LaTeX (e.g. for shadow maps paper).

• AsciiDoctor: I used this for modern Pascal introduction, sources on GitHub.

  This is very comfortable for editing, markup language is wiki-like (but can be processed reliably by more tools, as AsciiDoctor is more standardized than Markdown).

  The output is customizable, so it can be made to fit within a website.

• Jekyll (I used it for PasDoc website):

  + excellent version control integration, you commit markup (e.g. using Markdown or AsciiDoctor) to version control

  + secure: you just generate HTML, and put it on your website. No need for server-side or client-side scripting, when you can just put HTMLs on your website, when all you need is a static website.

• News using WordPress: WordPress is good, esp. I like

  + customizable shortcodes (e.g. to solve the “easy way to add API docs links”)

  + easy way to edit news

  + easy way to upload images

  My main concern of doing 100% content in WordPress is that you don’t have version-control support. While WordPress has versioning, it is nowhere as comfortable as “just a bunch of text files in a GIT repo”.

  + we want to use existing templates for WordPress sites. So using WordPress is beneficial to be able to use existing themes.

• GitHub wiki:

  + easy to use mark-up language, as above

  – lack of ways to customize it, looks like alien when mixed with our website

  – cannot add images, you have to upload them to normal website manually anyway. So adding images is least comfortable in this case, which makes it a “no-go” for our documentation that should have lots of screenshots.

  – lack of http redirects

• Our website using hand-crafted PHP with theme based on Bootstrap, which is our current main way to write website:

  + 100% flexible

  + PHP performed some part of “customizable shortcodes” that we need

  + looks native, we do what we want, using common theme, manual headers, automatic ToCs etc.

  – still some work to add new images, this is tiresome when writing documentation with lots of screenshots. You put them in htdocs/images/original_size, run make, add PHP/HTML code to include them.

We need to make improvements:

1. easy uploading images to current cge-www images/ , autogenerate their thumbnails

   This can be extension of current PHP code.

   A tag that adds images, and allows to upload image also in special website mode.

2. easy shortcode to add API docs.

   This can be extension of current PHP code (apidoc can be easier, to take auto-generated PHP table of Pascal identifiers). We already have PHP function used like <?php api_link('TCastleScene', 'CastleScene.TCastleScene.html'); ?>, we need to make it less tiresome, so that I can write just <?php api_link('TCastleScene'); ?> and it will find in what unit it is.

   Then it can be a WordPress shortcode too.

3. One markup, comfortable. Also replace GitHub wiki with it.

   My choice: AsciiDoctor, this proved right for modern Pascal introduction and later for
   PasDoc website. The fact that Khronos decided to use it for glTF specification is an outside confirmation that it’s good.

   Markdown can be converted to it using https://github.com/asciidoctor/kramdown-asciidoc + some manual tweaking for one-time conversion.

I hope you find these thoughts useful 🙂 These plans have been followed by actions — see section Documentation upgrades done lately above.

Please support the engine development on our Patreon. It funds development of the engine, including the efforts to document it perfectly.

Thanks to the related Lazarus issue being fixed by Juha Manninen, our editor can now open Pascal files in the proper Lazarus project reliably if you use Lazarus >= 2.2.

We also query Lazarus version, to activate new behavior only for new Lazarus. You should see your Lazarus version in the editor preferences window.

I have just released a new version of glplotter, a small utility implemented using Castle Game Engine to draw graphs. Graphs can be defined using an intuitive mathematical expression syntax (thanks to Castle Script). Multiple graphs are visible at the same time, and you can inspect the function graphs by moving, zooming, toggling various grids and scales.

Everything is documented on glplotter webpage. The source code is on the project’s GitHub page. The released binaries are on GitHub releases.

Please support the engine development on our Patreon. It funds development of the engine, along with all the extra tools.

With many thanks to Trung Le (Kagamma), we have a new experimental way to manage editor windows. You can freely rearrange them within the main form, you can rearrange them in tabs, you can also detach them from the main form. This is great to spread the editor windows across multiple monitors too, to e.g. have a large “Design” (game preview) on one monitor, and the rest (object inspector, hierarchy etc.) on another.

Usage:

1. This feature is experimental for now, and by default hidden. To enable it, create a file called enable-docking.txt in the editor user preferences directory: $HOME/.config/castle-editor/enable-docking.txt on Unix, c:/Users/USERNAME/AppData/Local/castle-editor/enable-docking.txt on Windows. If you ever used the editor, you will find existing castle-editor.conf in the same directory already.

2. Enable it by checking “Window -> Docking” menu item.

3. Close and reopen the project (you can also just close and reopen the entire editor).

4. Have fun! Drag the headers to rearrange windows. Drop them in new areas to anchor there. Drop them over existing areas, to add to tabs. Drag them outside of the main window, to detach them.

5. The layout is automatically saved (in layout.dock-layout in your preferences directory).

6. If you ever lose a particular window, you can invoke it by various “Window -> Xxx” commands, like “Window -> Properties”.

7. If you want to reset the layout, use “Window -> Reset Default Docking Layout”

The feature is very powerful.

We are waiting for champions to solve some of the issues (critical ones: 349, 348, 347, 345) and then of course the additional step to create enable-docking.txt will disappear. The additional step to enable them using the menu “Window -> Docking” may disappear too. All it takes is making it look and work perfectly, out-of-the-box, so that we can enable it for everyone without fear that new users could easily see some bug, or get easily into a weird layout without knowing how to “get back”.

Please support the engine development on our Patreon. It funds development of the engine, with many new features and fixes!

As we extend the engine, we sometimes have to deprecate some old features, that were not very useful, and sometimes prevented the engine API from being flexible and easy to understand.

One of these things was TCastleSceneManager class (renamed to TCastleViewport since 2 years), with its single “central” instance in TCastleWindow.SceneManager or TCastleControl.SceneManager. In a hindsight, I know it was a mistake to add such “central” reference to “one main scene manager”. It was much better to focus on adding multiple UI controls to your window (which we did), and realizing that various scenarios are possible:

• Some games don’t need TCastleSceneManager / TCastleViewport at all.

  E.g. applications with simple UI, where everything is just a simple 2D UI control.

• Some games need one instance of TCastleSceneManager / TCastleViewport (e.g. typical 3D FPS games). But it still may not exist always, so it better exists only within particular TUIState.

• Some games need more instances of TCastleSceneManager / TCastleViewport. Maybe on one TUIState (e.g. split-screen local multiplayer games), maybe spread in many TUIState instances (e.g. 3D game view, 3D map view etc.).

This observation led to the API you now have and use. But… the TCastleWindow.SceneManager and TCastleControl.SceneManager are still defined, for backward compatibility. And we advise everyone to use classes with Base suffix, TCastleWindowBase and TCastleControlBase, to avoid having this “automatic central scene manager” present. This is causing complicated engine code (for backward compatibility), complicated docs (why the Base suffix?), and unnecessary dependencies in CastleWindow unit (that right now has to depend on almost all CGE code, as TCastleSceneManager / TCastleViewport depends on TCastleScene that depends on almost everything else).

We had TCastleWindow.SceneManager and TCastleControl.SceneManager deprecated for 2 years now, since December 2019. Time to actually remove them 🙂 So, a plea: if you use TCastleWindow or TCastleControl, if you see messages that they are deprecated but you keep using them anyway… stop 🙂

The best way to upgrade is to switch to using TUIState, designed by CGE editor, and put TCastleViewport in your games this way. But as a last resort, you can even create local copy of existing TCastleWindow code from castlewindow.pas, it isn’t long.

At the end of January 2022 (in 1 month) we will actually remove TCastleWindow.SceneManager and TCastleControl.SceneManager. The TCastleWindow and TCastleControl will become equivalent to their ...Base counterparts, and we will encourage to use again the simpler names (as there will be no point in using ...Base suffix). It is easy to try it right now: go to src/common_includes/castleconf.inc, find

{$ifdef FPC}
  {$define CASTLE_DEPRECATED_WINDOW_CLASSES}
{$endif}

and just remove it. Or add something, e.g. dot, to make the {$define CASTLE_DEPRECATED_WINDOW_CLASSES} inactive, like this:

{$ifdef FPC}
  {.$define CASTLE_DEPRECATED_WINDOW_CLASSES}
{$endif}

This will change the TCastleWindow and TCastleControl to be equivalent to their ...Base counterparts, without any deprecated SceneManager property.

Please support the engine development on our Patreon. It funds development of the engine, it means we have resources to design the engine the right way!

Blending is a common technique to achieve partial-transparency of various things (UI, 2D and 3D models). In many cases it means that your texture has an additional channel, alpha channel, that determines how much each pixel is opaque.

But there is a trap — colors from fully transparent pixels (that you, by default, don’t see in usual 2D image viewers and editors) can still affect the final rendering output, because when rendering we do filtering that can average RGB colors from neighboring pixels. The way to fix these problems is alpha bleeding, which means that your transparent pixels also need some sensible RGB values.

I have written now a proper manual page about the “alpha bleeding”, what problem does it solve and how to actually use it.

Our castle-view-image, that is already bundled with CGE binary release, contains a ready menu item to do it.

Note that the issue is not specific to Castle Game Engine. It’s an effect of how the GPU is using alpha and averaging RGBA pixels, and it really affects all game engines that exist. But some engines hide it better than us right now 🙂 E.g. Unity does alpha bleeding for you automatically, during texture import, if you select Alpha is Transparency. In CGE, we do not (yet) have so trivial UI to do this.

Please support the engine development on our Patreon. It funds development of the engine, it means we have resources to add engine features that make it easier to use and we have time to document them properly!

Thanks to Eugene Loza we have an improved pipeline for compressing textures using ASTC (useful on modern Android and iOS versions). See Auto-generated compressed and scaled textures about GPU texture compression in general.

Now:

• We can use astcenc tool and the .astc file format it generates for ASTC textures.

  See PR #321 for details.

• We even package the astcenc tool within the Castle Game Engine. We can, as it is open-source. It’s available in our binary releases as well as GitHub Repo (tools/contrib/). So you don’t have to download/install anything to be able to compress textures using ASTC.

  See PR #351 for details.

• Where to go from here? What more texture tools to “just bundle with CGE”? I started the thread on our forum. I think bundling AMD Compressonator with CGE is sensible.

Please support the engine development on our Patreon. It funds development of the engine, it means we have resources to do the engine in the right way — that works efficiently and is easy to use.

Thanks to PR #354 by Andrzej Kilijański dragging-and-dropping items in the editor is now better:

1. We do not reload hierarchy. This is esp. important when editing larger hierarchy of UI / transformations.

2. We keep the selection.

3. Also occasional Access Violation when doing drag-and-drop on hierarchy on Windows is fixed. See the issue #353.

Please support the engine development on our Patreon. It allows us to fund development of important features and fixes such as this. Andrzej’s time is financed by Patreon proceeds and we have much better engine thanks to it!

We now can use Vampyre Imaging Library to load and save images in various formats. After some testing, I’m very impressed with how easy it is to use Vampyre, and we are already:

• Using it with Delphi by default (no need to do anything).

• Vampyre source code is bundled with CGE, so you don’t need to set up / download anything extra. Vampyre is just automatically used under the hood by CGE to load/save some image formats. It is in src/vampyre_imaginglib of CGE, and CGE build tool automatically passes paths to it when building CGE applications.

For FPC:

I have also tested using Vampyre with FPC, and our build tool already contains the necessary paths. Every FPC user of CGE can just define USE_VAMPYRE_IMAGING symbol to use it. There are a number of ways to define it, the advised is to define it in CastleEngineManifest.xml, like here.

In the near future we will likely just use Vampyre with FPC by default, so the need to define USE_VAMPYRE_IMAGING is just a temporary thing. We just need more testing, with various FPC versions and platforms (see lower for examples).

In the more distant future, we may also remove the code to use FpImage with FPC, even as a fallback, since Vampyre seems just better than FpImage in every way.

Advantages of using Vampyre:

• Vampyre Imaging Library is cross-platform.

• Works with both FPC and Delphi. Unlike FPC-specific FpImage or Delphi-specific PngImage (and other stuff in Delphi Vcl.Imaging).

• Supports many image formats, both reading and writing. It even supports a bit more than FpImage, e.g. we added now loading TIFF, JPEG 2000, XPM without any extra tools.

• It has really good and simple API. Almost entire CGE + Vampyre integration code is in this include file.

• It is well documented on https://imaginglib.sourceforge.io/ , with manual and reference.

• It is efficient. Unlike e.g. FpImage from FPC — which turned out to be very inefficient when it comes e.g. to PNG image reading (despite our efforts to make it fast, using TFPCompactImgRGBA8Bit, using format detection in InternalDetectClassPNG).

• It doesn’t require any external libraries. Unlike LibPng approach in CGE.

Help us test!

I’m eager to hear about your tests of Vampyre on various platforms, with various compilers, with your images.

You can use e.g. our castle-view-image (image viewer) for easy testing. It compiles now with both FPC and Delphi.

You can also use a new cross-platform example: examples/images_videos/image_display . This loads the image in a number of formats (making sure we support them all), and can perform speed testing – to compare loading times of various libraries, so you can verify the speed of Vampyre vs FpImage vs LibPng yourself.

What about LibPng, DDS, KTX?

Vampyre is still a bit slower for reading PNGs than LibPng (about 2x slower on Linux/x86_64, which is anyway better than FpImage that was 4x slower), so we will keep relying on LibPng for PNG. Still, Vampyre is a much better fallback than FpImage when LibPng will be missing.

We may rely on Vampyre for some formats that are currently built-in in CGE code, too. This includes BMP, RGBE, PPM.

We will likely keep our own CGE code for loading DDS and KTX though. (Vampyre can handle DDS. But we want DDS and KTX to go though a similar code path in CGE, and we want to efficiently use all their features — cubemaps, texture compression, 3d textures, mipmaps.)

I have just merged a giant pull request to CGE with 334 commits! This is a big work from Andrzej Kilijański and me (Michalis Kamburelis) from the last 3.5 months, and it includes:

• Delphi compatibility! See PR 350 for details what works with Delphi — in short, almost everything 🙂 The supported compilers page was already updated. See Michalis presentation on DelphiCon 2021 for a great CGE overview for Delphi devs.

• Integration with Vampyre Imaging Library. For Delphi it is even active by default, for FPC it soon will be. I will write about Vampyre Imaging Library more in upcoming news, but basically it is a great, open-source library for reading and writing images. Cross-platform, cross-compiler (both Delphi and FPC work flawlessly), efficient. We will rely on it more and more in CGE.

• Upgrades / big rework of numerous examples. To be compatible with Delphi, but also to be designed using editor, use TUIState, be cross-platform, have a nice README… Of particular interest are these new examples:

  • examples/images_videos/image_display (includes a speed test, to compare various image-reading libs)
  • examples/curves/use_designed_curve
  • examples/3d_rendering_processing/shader_override
  • examples/3d_rendering_processing/multiple_viewports (big rework of older example)
  • examples/3d_rendering_processing/listen_on_x3d_events (big rework of older example)

• Inspector (announced previously) updates:

  • work on mobile (press 3 fingers for 1 second to toggle),
  • various fixes (including to show “Swap” time on Windows correctly),
  • default shortcut changed to F8 (to not conflict with Windows debugger in Delphi; I will update the video about it soon),
  • customizable by Container.InputInspector (instance of TInputInspector).

• TCastleNotifications.DesignTestMessagesTimeout – cool feature to preview how notifications look like in design mode.

• KTX fixes to handle RGB8, SRGB8 colors.

• We officially deprecate dynamic library to read and render 3D models using CGE due to low usage. It was never the right way to use CGE (since it exposes a tiny API compared to what you can do with real CGE). We also deprecate some examples, as they no longer present “the advised way” of doing various things.

• Fixed confusing LibPng warnings on Android (to account for the fact that LibPng is initialized on Android later).

Edit: I initially wrote that it is “250 commits”, not 334. It turns out that GitHub PR interface just shows only 250 commits, but there are more! 🙂