These are the jots I wrote down as I learned my way through AppSheet. So this a messy and not so coherent post. Info is scattered here, not necessarily in a sensible order. This is not a tutorial.

My interest comes from the fact that I’m looking for an apartment, and I want to go through a lot of details in each apartment I visit. So I looked for a tool that allows me to organize the information I collect, from the ad, through the thoughts I have about the apartment before visiting it, and most importantly, to efficiently examine the apartment.

This can be done with a plain paper form, of course, but when there are many details and stages, it’s easy to miss one. So the crucial part of my application is to get a quick final view that ensures me I’ve done everything, just before thanking for the visit. This is the most important part, and it’s also the most difficult to implement.

In hindsight, it took 9 days, which is much more time and effort than I intended and wanted to invest on this. I had the “almost done” feeling more or less from the beginning, but every next step turned out much more difficult (and annoying) than I imagined. AppSheet makes you feel like a code monkey that doesn’t know coding. And if AppSheet is used for a long-term project that is maintained over time, be sure to have a QA team that tests everything in every possible combination. The framework invites silly bugs everywhere.

General

• In a nutshell, AppSheet takes data from a database (preferably represented as a spreadsheet in Google Sheets) and presents this data in different Views, allowing both displaying and changing the data. The other side of AppSheet is called Automation, its ability to generate files, send emails / SMSes, submit HTTP requests etc. with the information in the database, based upon a template. This can be a pdf for an invoice, or a summary report of an inspection. Or an XML or JSON file, pushed to a web server to initiate some action on the information. So Views are primarily intended to feed data into the system, and Automation covers the consumption of the data for a useful purpose.
• AppSheet at no cost is unlimited in time and covers every feature of the platform with up to 10 test users. One user, myself, is enough. In free mode, the app can is limited to Prototype status, which has no significance when used only by myself.
• AppSheet reminds of Drupal in many ways, and it’s not necessarily a good thing. Lots of click-click-click around, everything is graphical, and one ends up feeling like a machine doing a silly task a gazillion times because it’s the same as configuring a Windows Server. Every. Single. Detail. Must. Be. Configured. Manually. With a prayer that there will never be a need to change all those settings that have been repeatedly configured. Plus, small bugs creep in everywhere, as there are a whole lot of details that one must keep track on when making every little change.
• Another thing common with Drupal is that every little task takes a lot of time, and sometimes the answer is simply “no, that’s impossible”. At least with Drupal, one can hack the PHP code. For example, the one thing that I considered important, but didn’t implement, was to get a list of the items in a checklist that I didn’t check. I didn’t want to implement the checklist as individual Yes/No (boolean) columns, because adding an item would require modifying the table, regenerating the schema, editing the View and then the report. So I did that with an EnumList, but the only information it emits is what was checked. So all in all, there is no way to implement this rather important feature without turning the checklist into a nightmare to maintain in the long run.
• There are certain limitations on databases and tables, in particular 1000 rows per database in the Free plan and for all plans there are 20 tables per database and 100 columns per table. Don’t worry, you’ll lose your sanity before getting even near these limits.
• Press the “Save” button to the upper right to make changes visible (e.g. when checking with mobile app). This also clears existing warnings, whether they were tended to or not.
• Offline use works great: No problem using the mobile phone even with no Intenet coverage. Update to cloud occurs later (may be required to open the app for that). Actually, the only reason I use AppSheets instead of a JavaScript home-cooked application is the ability to work offline.
• Backing up an app: Impossible. There’s no way to download the app’s own settings, and deleting an app accidentally is easy. The best solution I found for this is to share the app (or better, a copy of it) with another user (being yourself) and make a copy of the app as the other user (note to self: It’s the “obscura” account). And never log in as the other user again. This way, the copy at the other user is safe from accidental tampering. I speculate that the reason for not being able to download the app itself is that AppSheet’s business model is pay-per-user, and it’s free for less than 10 users. So had it been possible to download and restore the app from a file, a lot of no-cost single-user accounts would have been created and deployed.

Random insights

• Each time “Save” is clicked, a new version is saved. It’s possible to go back to older versions. Version history is saved 7 days back by default.
• It’s possible to run a preview of the app by right-clicking the three dots to the right of the app’s description in the list of apps. Doing that, the specific version of the app is previewed, ignoring subsequent new saved versions of the app. There’s a misleading “refresh” icon on the app preview. It probably only updates data, not the app itself.
• If a “Map” view is generated automatically, delete it. Otherwise it will show up as the default view for everything (push marketing, it’s called), and it’s as annoying as it’s pointless.

Sources of info

• List of functions for use in expressions. There isn’t so many of them.
• Boolean expression (and that “not equal” is <>, not !=).
• About Templates
• About expressions and built-in variables. For columns that are references, an expression like [colA].[ColB] is used (official docs about dereferencing).
• If-expressions in templates

Creating a new app

This describes how to create an app based upon a spreadsheet. The spreadsheet’s first row consists of the description of each column, and the following rows contain data. Multiple sheets are treated as separate tables, each sheet’s name being the AppSheet table name (so don’t rename the sheets in the spreadsheet).

Upload the .ods or .xlsx file to Google Drive, or start a new Google Sheet.

if you’re already registered with AppSheet, right-click the file in Google Drive, pick Open With and choose AppSheet. An app is generated automatically from the data in the file. The name of the app will be the same as the originating spreadsheet’s file name. The column types (text, number, price etc.) are automatically assigned, but can be changed later on in the Data section. It’s even possible to change the type to Image, in which case pictures that are uploaded or taken with the mobile phone are stored in a subdirectory to where the Google Sheet item is saved with a name like appname_Images/. The value in the spreadsheet is the path to the image file, e.g. appname_Images/J1.Silly.140258.jpg.

It’s best to delete the .ods or .xlsx from Google drive at this stage to avoid confusion: A Google Sheets entry is created in the same directory. Changes made with AppSheet go to this entry, not to the file used to create the app.

Doing the same from within AppSheet (not clear if this is better):

• Navigate to the page listing apps (AppSheet’s main page), click “+ Create”, pick “App” and then “Start with existing data”.
• Give the app a name. For category, I choose “Inspection & Surveys”, and then click “Choose your data”.
• Select an existing Google Sheets entry in your Google Drive account as the database (Google Sheets).
• Trying to use an .ods file as the data source caused AppSheet to hang (wait indefinitely to complete creating the app).

It’s also possible to use an AppSheet database for this purpose, but that seems to be a leftover from the days before AppSheet became part of Google. There’s no reason I’m aware of to prefer AppSheet database, and there’s no obvious way to make a backup of it, nor to restore. Possible with exporting to CSV with Automation, I suppose, but never tried it.

A directory named “appsheet” is created in Google Drives’ home directory. It contains a directory structure with an empty file called “empty.txt”. A subdirectory is created for each AppSheet app, and files generated by the App go into that subdirectory. It’s possible to select a different target for output files, however (see below).

When an app is created, you get an email with an invitation to use it or develop it.

To do after a while

After playing around with an app for a while, it’s a good idea to make a few fixes. Go to the Views section (click on the third icon from the top) and then on the settings icon (a flywheel).

• In Views > General, choose the starting View.
• Also in Views > General, set the Image Upload size to the desired level (High for 1600x900 with my phone, of Full).
• Be sure to have enabled “Allow drawing on images” on all Image columns (unless you don’t like that option).
• In Offline mode, enable “Store content for offline use”.
• Go do the Data view, choose the main table, and opt out deleting rows (the flywheel icon to the top right for a table’s settings). Delete rows directly on the spreadsheet if needed.
• Change the Default app folder to /theprojectname/appsheet-data or something like that, which should be a valid path in Google Drive. So that the files land somewhere sensible. This is done in the Setting section (flywheel icon to the left), under “Information”.

Views

• The View displays (or allows editing) a subset of column values of a row of a table. In some cases, several rows are listed in one View. Which columns are displayed, how they are displayed, and if they are editable, is what the View’s configuration is about. But no matter how you twist and turn it, a View shows values of columns (possibly along with the column names). If you want to display anything, turn it into a column (possibly a Virtual column, defined only inside AppSheet, see below).
• The relation between the AppSheet’s table, which is used in Views and expressions, is not 1:1 with the table in the spreadsheet: Virtual columns are added, either by AppSheet itself (e.g. _RowNumber and Row ID), or by the user. In the latter case, the value of the Virtual Column is an expression that the user supplies. This can be a constant, for example if the purpose is to create a menu button in a View. The expression can also be a formula, which depends on the value of other columns of the same or other tables. In these expressions, [X] is the value of column X of the same row.
• Clicking / tapping on an displayed item on a View initiates an Action, which often consists of opening a view for a specific row in a table for read-only access or editing. These are system-generated view with names like “main_detail”, “main_form” and “main_inline” for a table named “main”. These Views can be modified like any other view. And a whole lot of other actions can be added and used as the response.
• It’s often difficult to figure out which View appears in response to clicks, as it’s often a system-generated one. Be sure to have the Edit option on (top-right on the development screen). Hover over the relevant area, wait for the pencil icon to appear, click it and pick e.g. “Edit View”.
• If a system-generated view is deleted, it’s created anew after saving, apparently with default settings.
• If a column is added to the spreadsheet’s table, it’s not available for use in the AppSheet immediately. The AppSheet’s table schema needs to be regenerated for this to happen (click the Data icon, click the table, three dots, and then “Regenerate Schema”). Virtual columns are not deleted, despite the scary warning one gets.
• When clicking / tapping an item can result in an detailed view or the possibility to edit, these are system-generated views, that appear at the bottom left in the Views subsection. These Views can be modified.
• Views listed under “Primary Navigation” are accessible directly at the bottom part of the screen. Those under “Menu Navigation” are accessible through the menu. “Reference Views” are invisible to the user, except for when requested from other views, for example in an Action (i.e. a response to clicking / tapping an item).
• There are Format Rules allowing to change colors of items etc depending on boolean expressions. In most cases, they apply to columns and how they are displayed (with a different color, or with a colored icon added). Unfortunately, it’s impossible (as far as I know) to write catch-all rules for several columns, as the expression for activating the rule doesn’t play ball with the [_THIS] expression, which means “this column”. So for example, if you want a rule that marks unfilled columns with a red dot, add (or duplicate) one rule for each and every column, and be sure that the rule for column X doesn’t by mistake change the display format of column Y. It’s just an invitation to make silly bugs.
• The column’s name is displayed above the column’s value in forms and detail Views by default. This can be changed by opening the column’s settings in the Data section, and change the “Display name”. So use concise names in the spreadsheet.
• A column of type Show is useful for text, URLs, images etc, that appear in forms for instructions, setting page breaks etc. These are best added as virtual columns with literal values in the expression. Don’t forget to turn off the “Editable” attribute of this column. It’s quite unfortunate that instruction text appears as an extra column along with the spreadsheet’s data, but that’s the way it’s done.
• To navigate to another view as a result of clicking on a column (possibly a Virtual Column acting as a menu entry):
  • First, an Action needs to be created. Click on the Action icon (looks like electric zap), click on the “+” at the top left, and add an action
  • Pick the table related to the view that this action shall work on, and pick “App: go to another view within this app”.
  • As for the Target, choose the expression LINKTOVIEW([targetview]) (given that “targetview” is the name of the column containing the name of the view to navigate to, as a plain text column).
  • Go to the relevant item in the View, and choose the newly created action under the Behavior section.
  • If the item is a Virtual Column just for the purpose of being a menu item, change its Display Name to ” ” (a normal white space inside quotes) so that the column’s name doesn’t appear on the menu, which is both ugly and consumes space. Choosing an empty string, “”, has no effect.

There are several types of Views, but the most important for a plain spreadsheet-based app are:

• Form: All columns listed in the “Column order” are shown and editable. The title of each input (text box, drop-down menu etc.) is the name of the column in the database or spreadsheet, unless overridden in the column’s configuration (in the Data section). Unfortunately, a back-swipe on a mobile phone is interpreted as “Cancel”, and there’s no way around this. One must explicitly tap on Save, or else all is info is lost.
• Details: Like Form, but read-only, with an icon at the bottom right for editing the content, which switches to a Form with the same set of fields. Unfortunately, blank fields are not shown unless their Show property is set to the expression TRUE (just checking the checkbox isn’t good enough, it has to be done with an expression).
• Table: As the name implies, like Details, but with the data shown as a table (with rows being rows, columns being columns). Selecting a row brings to a Details view of it. It’s possible to configure which columns appear in the table, possibly only one. So this can be a concise way to display a list of rows. It appears like the column marked as Label won’t appear in a Table View even if chosen to do so. Why is unclear.
• Card and Deck: Each row gets a small pane with two or a few selected column values shown. Selecting one brings to a Details view.
• Gallery: Shows the image associated with each row (if such exists) and the value of the column marked as Label.
• Dashboard: A View containing other Views.

Adding a new table

• In Appsheet, click the Data icon (the second icon from the top).
• Click on the “+” (Add new data)
• Navigate to the relevant Google Sheet.
• Navigate to the related table, and add it.

Settings for each column in a table

In the Data section, there are several attributes one should look at in a table’s configuration for a table:

• Name: Must match the name in the spreadsheet’s header (or the column name of a database’s table). The app fails to load otherwise.
• Type: Number, text, image, reference, price, there are a lot of options. The automatic choice is usually correct for the simple textual columns. For more sophisticated input (drop-down menus, images, references etc.) this is the place to configure that.
• Key: This checkbox is checked for one column only, selecting the key column in the database sense. Relevant in particular when using references.
• Label: This checkbox is checked for one column only (plus, possibly, an image), selecting which column appears in several views as the row’s main description.
• Formula: Left blank for columns taken from databases, must contain something for a Virtual Column. When set to something, the value of the column is the expression in this formula.
• SHOW?: As its name implies, it controls if the column should be shown in views. This checkbox should usually be checked: If a column isn’t desired in a view, it should be removed from there. If the column should be displayed in Details Views even when blank, checking this isn’t enough: The Show propery must be set to TRUE as an expression.
• EDITABLE?: Can the value of the column change?
• REQUIRE?: Must the column have a value to allow finishing an edit session containing this column?
• Initial value: The column’s initial value when a new row is created.
• Display name: If left blank, the Name field from above appears above the value of this column in forms etc. Display name overrides this otherwise.
• Description: For internal documentation, and is also used instead of the column name if “Display Name” is left blank.
• Search: Is this column involved in searches? This is a tricky one: Enabling this means that the row will appear in searches even if the column isn’t displayed in the View, and vice versa: Even if a column is displayed, the search doesn’t take it into consideration unless the Search option is enabled for the column. It would, of course, had made more sense to define this option per View, but that’s not the way it works, unfortunately.
• Scan, NFC, PII: Can this column accept data from these input methods?(scan means barcode scanning)

Modifying the spreadsheet

It’s possible to modify the spreadsheet even after the app has been created, including adding a column in the middle. The immediate effect is an error (“unable to fetch app definition” on the mobile app, or just an error on the development console). To resolve this, pick the “Data” icon (second from the top”) on the development console, and then click the round arrow (as in “reload”) to regenerate the structure. A scary warning pops up in response to that, but just go ahead.

Note that the connection between AppSheet and the spreadsheet is through the names of the columns as given in the first row, as well as the name of the sheet. These should not be modified (unless the manipulation is intentional).

References etc.

AppSheet definitely supports relational databases. In fact, by choosing names of columns where one is the plural form of the other (“owner” column and “owners” table) the relation is possibly set up automatically. This is however not necessarily a good idea, because the tools choose which column’s value identifies the entire row — and getting this wrong could mess up things later, and it’s difficult to fix it afterwards.

Rather, if a spreadsheet (or virtual) column (in “table A”) is assigned the type “Ref”, it’s also required to assign the table referred to (“table B”). The value of the column selects the row that has a key column equal to it. In the View showing this column, the label column of the referred table is shown. Clicking / tapping on the label leads to a View of that row: its columns’ values as well as other rows referencing it. In order to access a specific column, add a virtual column to Table A with a formula like [the_refcol_in_table_A].[the_col_in_table_B]. It’s not as difficult as is sounds, as the GUI starts suggesting completions when it sees the brackets and the dot.

Note that in Appsheet, a reference actually means the whole row. In many practical situations, only the label column is displayed, but it’s important to remember that conceptually, a Ref column represents the entire row it refers to.

And this brings me to back-reference. When a table B is referenced by a column somewhere in table A, a virtual column is automatically added to table B. This column has the type List and its value is something like REF_ROWS(“table A”, “referring_column”). The arguments are the name of table A and the name of the column with type Ref in that table. So this column lists all rows in table A that refer to the row in table B. Consider deselecting its “SHOW” checkbox in the Data view, if this back and forth is undesired.

Also, note that the back-reference is shown as a list of inline elements on the View of Table B. In order to determine how it appears, configure the system-generated view with name e.g. tableA_Inline (for example, if it’s a table or a deck). Even more important, it allows choosing what happens when one of the elements is clicked: View details (e.g. tableA_Detail view), edit (e.g. tableA_Form) or do something completely different with an Action?

And when references exist, it’s important to get the keys right. It’s possible to choose anything that is guaranteed to be unique, but because references relate to the key, there might be weird side-effects if the value of the key column changes. For example, if we choose a person’s email address as the key, all references to that row are lost if this person changes this address.

What we really want is a solid reference to the row itself. This is best done by adding a dedicated column for that in the table (i.e. in the spreadsheet), and assign it with a unique value when a new row is created: Use UNIQUEID() as this column’s initial value. When AppSheet adds a new row, it will calculate a unique ID and put the value there. Make this column non-editable (possibly also turn off “SHOW” too). In fact, AppSheet does this automatically when I give a column the name “key_id”.

Note that calculating UNIQUEID() in a virtual column is pointless. That’s not a solution for creating a unique ID for a row.

Other insights:

• Allowing upload of multiple images (or adding other items): Create a table consisting of “key_id” (used as key), “ref”, “image” and “weight” columns. The “ref” column is declared as a Ref type to the table in which the images will be stored. Enable “is part of”, which makes it possible to add new items from the main view. So all in all, there is a list of images in the main view, (of table B). Each row in the view points to a row in table A. This makes it possible to add an arbitrary number of items from a view of a row in the main view. Each item becomes a new row in table A, with a reference to table B. “weight” is a number, allowing to control the order when displaying the image. Using “key_id” as the key, and not “image” (which is the name to the file containing the image) makes it possible to show the same image in different Views, even from different tables.
• Drop-down menus with text taken from somewhere else: Prepare a table (a sheet, and import it as a table) with keys as numbers and text (or anything else) as values. Then, on the table where we want the drop-down, set the column’s type to Ref, and point to the said tables as the reference. Select the input type to Buttons or Drop-down. In order to allow adding new possible values, set it to Enum (or EnumList for multiple values), and the Base type to Ref. Choose the said table as the target for the reference. Note however that in the latter method, all already existing keys must be added manually to the column’s definitions, exactly like any Enum type. The only difference is that the keys (possibly numbers) are fed manually, not the text. So if adding new possibilities is required, plain Enum is probably the best way.
• It’s impossible to get a list of options not chosen in an EnumList. It’s possible to write an expression that reduces the chosen elements from a given list with those selected by an EnumList, but that requires keeping this expression and the EnumList’s value in sync. And if one forgets to update the expression after adding it to the EnumList, it’s a nasty bug.

One long row or relational database?

As there are a lot of pieces of information about an apartment, which can be divided into categories and will be handled by different Views, my database manager self was inclined towards a relational database approach. One row with a few columns for each apartment in the “main table”, and then several tables, one for each information category (i.e. View), referenced by the “main table”.

The alternative approach is one table for all information (except for images and other utilities) and to make one long row with all info. It’s up to the Views to select which columns to show: All columns in the original table (or spreadsheet sheet) don’t have to be visible. It’s possible to make several views of the same table. But this means a lot of columns to handle (add virtual columns to that), making it difficult to keep track of the whole thing.

I went for the one long row approach. The main reason was that this approach makes it easier to reorganize the Views if needed. For example, if fields are added over time, and a View becomes too crowded, it’s not a big deal to split it into two Views, if they both refer to the same table anyhow. Or to move a field to another View.

That said, the separate table table approach would definitely have worked: When a new item is added, that means a row in the main table. A new row is created on each separate table by virtue of a menu button (a virtual column with a specified Action). LINKTOFORM() allows opening a form with specific initial values, so the key column can be set in the child table to the parent table’s key column value, ensuring a 1:1 relationship (and prevents duplicate rows).

For reports, the child table is shown as a back-reference to the main tables with a Start/End pair of template tags (maybe there’s a simpler way to do this).

In hindsight, I should have taken the separate table approach, but in a way I didn’t think about at first: There should have been a table with one row for each room, regardless of its type. This would have reduced the number of columns in the main table, and the amount of work and headache is linear to the number of columns. The trick would have been not to show all columns of a room’s table in the form View, but only those suitable for the specific type of room. The rules for which types of rooms exist, and which columns should be shown in the View would be listed in a separate table. Hence the “Show if” rule would look up the column in this rule table, in relation to the room’s type.

Use slices?

The question is how to display a subset of the columns in different Views.

The seemingly natural (and less preferred) way is by using slices, which is a virtual table, calculated on the fly, consisting of a subset of rows and columns as configured. This allows displaying only a subset of rows, columns and actions, filtered with a condition on the data (or just not show all columns). Slices are configured with the “+” icon on each data set (inside the “Data” section) as they are considered some form of table. Alternatively, slices are hinted on the View’s configuration as well.

This would have been the preferred way, had it not been for an annoying caveat: If a new column is added to the database (i.e. sheet), it appears in all slices. This makes slices a no-go for the purpose of selecting columns for a view.

Instead, the columns to display should be selected in the “Column order” part, which can be found in the “View Options” section for each View configuration. Note that “Column order” isn’t present for Views that don’t display columns, e.g. Deck and Gallery.

It makes sense to start from generating an empty view, and copying it for each time a new view is created.

Creating a pdf file from the data of a row

…or for that matter, a CSV / JSON / XML / HTML / XLSX file.

The important point is that this is done with Automation, and that the execution is triggered by an Event, and not as an Action. In other words, clicking something on a View won’t trigger the generation of the report directly. Rather, clicking the entry in the View causes an Action that modifies a row in a table, or to add a row a table, and that, in turn, triggers the Event.

There is no direct connection between the Action and the task it requests — the generation of a file, in this case. It can take 20-30 seconds from the triggering event until the file is created.

One option to request the generation of a file from an app: First, add two columns to the table for which the report is desired: “do_report” and “last_report”. In AppSheet, regenerate the schema if necessary, and set the columns as follows:

• do_report: Type is “Yes/No” (sweet talk for Boolean), turn off “show”, initial value “FALSE”
• last_report: Type is “Text”, and set Initial Value to “Never”. “Editable” must remain enabled, even though it shouldn’t be accessible by forms: Its intention is to be set by the Bot, and this is possible only if the column is editable.
• Add a virtual column named “generate_report”, “Text” type, with expression as follows:

  CONCATENATE("Generate report
  Last report: ", [last_report])

  Note that there is a new line in the middle of the string.

• Create a new Action (Bzzt icon): For “Do this” pick “Data: set the values of some columns in this row”. Pick the “do_report” column and set the expression to TRUE. I’m going to use the Virtual Column defined above as a menu item, so for Position, pick “Inline” and at “Attach to column”, pick “generate_report”. In Behavior, set “Only if this condition is true” NOT([do_report]), and optionally also enable the request for confirmation.
• Add the “generate_report” column to the View (or Views) from which the report should be requested.
• Optionally (and recommended), add a Format rule for generate_report. The condition is [do_report] and if so, the text should be grey (color #888888) and in italics. For “Format these columns and actions” pick generate_report, and possibly also the action created just above. Without this, there is no immediate visual feedback to “pushing the button”.

Now to the part that actually generates the report.

• Go to the Automation section (robot icon) and create a new Bot.
• Click “Configure Event” and give it a name.
• In the pane to the right, the Event source should be “App”. Pick the table from which the Event is triggered.
• Select only “Updates” for Data change type.
• Condition: [do_report]
• Next, define what should be executed — the Process. Add Steps, which is more or less like a line in a computer program: Each of them can perform something, it can check a condition and branch, or wait for the condition to be true. Or call a process. These are steps to define (give each some name, as usual) and configure them on the pane to the right.
  • Type: “Run a data action”. In the right pane, pick “Set row values”, and set “do_report” to FALSE.
  • Type: “Run a task”. Set up the task as follows in the right pane:
    • Pick Create a new file
    • HTTP Content Type: pdf
    • Template: Click Create (or pick an existing template, if you have one from previous sessions). A “View” button appears after this is completed. More on templates in a separate section below.
    • Optional: Change File Folder Path to “/” (with the quotes), File Name Prefix to “report” (once again, with the quotes) and opt-in Disable Timestamp. This way, report.pdf is created at the project’s root directory (not Google Drive’s root directory in most cases). Expressions can be used for these, of course.
  • Type: “Run a data action”. In the right pane, pick “Set row values”, and set last_report to TEXT(NOW(), “DD/MM/YYYY HH:MM:SS”) for British time format.

Note that it’s possible to create a pdf file, but the template file is an HTML file. This makes it much easier to handle the template, and it’s always possible to go back to the HTML file to see what went wrong. The disadvantage with an HTML template is that the file generation fails on the tiniest HTML syntax error. For example, a missing </tr> tag. Or even more ridiculous, a missing </p> tag.

In order to view the generated file by clicking a menu item in a View, create a Virtual Row for this purpose, and assign it with an Action when selected. This action should be “External: Open a file”. The “File” entry is a URL to where the file is stored on Google drive. Obtain this link with the Share feature, so the expression becomes something like “https://drive.google.com/file/d/1d8AWHjLJdK9cyhYGBPnuVJJ0asmLmZVw/view?usp=sharing” (the quotes included). This is not a security risk as one chooses to share only with oneself (when obtaining the link from Google Drive).

If the execution fails due to a malformed template file (something inside the markups didn’t work fine), the app’s View doesn’t always get updated until a restart of the app (or reload the entire page on a web browser).

Actually, the way the whole thing behaves is that the menu button gets greyed out immediately after being pressed, and then gets back to normal after the file has been generated. But the color is determined by do_report’s status, which is turned back to FALSE before the attempt to generate a file. So if the file generation takes time, it may return to normal before the file is updated — but this has never had any practical consequence for me. But why doesn’t it go back to normal when the file generation fails? do_report is FALSE either way.

To view error messages: Go to the “Monitor” section (icon at the left side, one from the bottom), and pick “Launch Log Analyzer” in the “Audit History” part. After some graphs, there’s a table with the last tasks initiated. Where there is an error, click the binocular icon, for details. A JSON text entry appears. The “Errors” key is where the error message is shown. It can take a few minutes before the entry with the error is shown, and before than, there may be entries showing success even if an error has occurred. So wait a few minutes before ruling out a failed run.

Templates

A template is a file in Doc or HTML format. Everything is passed through to the output document, except for markups with the <<expression>> form (or, for an HTML template, &lt;&lt;expression&gt;&gt;. The expression is evaluated exactly in the same way as the value of a Virtual Column, and the markup is substituted with the expression’s result. Hence to obtain a substitution with the value of a column named “thecol”, just insert <<[thecol]>>.

No matter if the output is pdf or HTML, when using HTML templates, be sure that the HTML file has all the classic wrapper tags: <html>, <head>, <body> etc. or else AppSheet ignores all <<something>> (or actually, &lt;&lt;something&gt;&gt;) markups altogether and the output is the template file itself.

There are special kinds of markups:

• Start/End for looping on a list of keys to a table
• If/EndIf markups, allowing a segment to be included or not. Apparently not available with HTML templates, as it’s not listed explicitly, and I failed to use it (so I used the IF() function instead).

The automatically created template is simple and not necessarily very helpful: This template encompasses all real columns (but not the virtual ones), showing the column’s name (not its display name) and their plain values. Its most valuable part is where it shows the use of Start/End (when relevant, i.e. when there are back-references to the displayed table). Note that as shown in the template, the Start and End tags must be inside a <p> block. They can’t be just a floating piece of code, or within another couple of tags (<div> doesn’t work, for example). Otherwise one gets a misleading error message saying the table with name “Start:” can’t be found + a lot of error messages for everything between the two tags.

The template is annoying in particular in relation to Ref types: The actual value, which is the key itself, is shown for references, and not the referenced value, as seen on the app’s Views. It would have been sweet of AppSheet to look up which of the referenced table’s columns is displayed in the View, and show that. But there’s another reason for this: Say, that the Ref column is “choice” and the column to display on the referenced table is “label”. The expression for the value to display is [choice].[label]. But if “choice” happens to be blank, and this expression appears anywhere in the template, the file generation fails. So be sure to assign an initial value to all Ref columns when creating a new row. Plus, possibly make a simple sanity check on the column, just in case:

<<IF(ISBLANK([choice]), "Error!", [choice].[label])>>

Not sure if this is bullet-proof, but it solves the most obvious problem.

I’ve also got the impression that if the result of the expression of an AppSheet markup expression returns anything containing HTML markups (e.g. <b> and </b>) the file generation fails completely.

Notes to self

These are cryptic notes to myself, for my own apps’ housekeeping tasks.

First of all, the account I’ve shared stuff with is “obscura”.

Adding/removing a new item to a checklist

• Update the related checkcnt expression with the new number of items, so that the report says all is done when it really is.

Adding a new column

• Be sure to regenerate the schema of the related table in AppSheet.
• Possibly add (or duplicate) an “Unfilled X” Format rule (e.g. ISBLANK([streetaddress]) ), if it matters that the field is empty. Mark with Cyan dot, and don’t forget to set the affected column correctly.
• Set the SHOW property of the column to the expression TRUE if it’s supposed to appear in the detailed view, even if it’s blank. Just clicking on the “SHOW” checkbox isn’t good enough.
• Set the Display property to something friendly

The striked-out bullets were relevant when I thought about using Detail view as the summary of all. I used a pdf report now, so they’re not relevant.

Adding a new View Form

• There should be a status_* column in relation to this in the spreadsheet. Its setting inside AppSheet: Ref to const_group_status with Buttons (not “part of”), enable Show, Editable, Require, Initial Value is “0″ (with quotes), Display name is “Status”. The four checkboxes to the right are left unchecked.
• In the main table, add a Virtual Column with the same name as the Form View it relates to, e.g. view_base for status_base (so it’s easier to match between them later). Its type is Text, and its value is the relevant menu item’s description.
• Only now, create a new Form view. In Column order, pick Manual, select all, unselect those not required. Be sure to keep the relevant status_* column.
• Go to the “Actions” section (Bzzzt icon) and add a new action (actually, duplicate a similar one). Name it e.g. “Jump to base form”. For a record of “main”, Do this is “App: go to another view within this app”. Set Target to LINKTOROW([key_id], “view_base”) where “view_base” is the name of the View. Set Position to Inline, and Attach to column as the related Virtual Column. Pick a suitable icon too. Unfortunately, there is no catch-all action for navigating.
• Duplicate a couple of “Mark incomplete” format rules, and adapt them to the new View: Change the expression to match the correct status_* column, and also the “Format these columns” to the relevant “view_*” with the same name. And test it. There’s no way around a couple of rules for each View.

Running Linux Mint 22.2 (based upon Ubuntu 24.04), and having a git repository on root filesystem to keep track of the computer’s configuration, the vast majority of directories are ignored. One of the is /lib, however /lib/systemd/ should not be ignored, as it contains crucial files for the system’s configuration.

On other distributions, the relevant part in .gitignore usually goes:

[ ... ]
bin/
boot/
dev/
home/
lib/*
!lib/systemd/
lib64/
lib32/
libx32/
lost+found/
media/
mnt/
opt/
proc/
root/
run/
sbin/
[ ... ]

So lib/ isn’t ignored as a directory, but all its content, including subdirectories is. That allows for un-ignoring lib/systemd/ on the following row. That’s why lib/ isn’t ignore-listed like the other ones.

But on Linux Mint 22.2, /lib is a symbolic link to /usr/lib. And since git treats a symbolic link just like a file, /lib/systemd/ is treated as /usr/lib/systemd. Ignoring /lib as a directory has no effect, and un-ignoring /lib/systemd has no effect, because to git, this directory doesn’t even exist.

So go

$ man gitignore

and try to figure out what to do. It’s quite difficult actually, but it boils down to this:

usr/*
!usr/lib/
usr/lib/*
!usr/lib/systemd/

It’s a bit tangled, but the point is that /usr/lib is un-ignored, then all its files are ignored, and then /usr/lib/systemd is un-ignored.

The only good part about this solution is that it works.

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
</head>
<body>
<div style="height: 1350px; display: flex; flex-direction: column; break-inside: avoid; border:1px solid #668;">
This is just a test.
<div style="margin-top: auto;">
This is a footnote
</div>
</div>
</body>
</html>

So how does it work? The important part is the “style” attribute of the outer <div> tag:

• height: 1350px: This sets the <div> block’s height to a full A4 page. Why 1350 pixels? I don’t know. I just tweaked with this figure until it got right. It’s possible another figure is needed on a different version of wkhtmltopdf. I’ve tried to set this with cm as well as pt units, but none corresponded to the standard figures for an A4 page. So I went with pixels, which clarifies that it’s a wild guess.
• display: flex; flex-direction: column: This turns this <div> block into a Flexbox container, with vertical packing. This is needed to push the footnote’s block to the bottom.
• break-inside: avoid: This tells wkhtmltopdf to avoid page breaks in the middle of the block. This makes no difference for a single page, but if this <div> block is repeated, this style attribute ensures that each block gets a separate page (unless any of the pages exceeds a page’s height).
• border:1px solid #668: This generates a border around the <div> block’s occupied area. Used only for finding the correct height attribute, and should should be removed afterwards (unless this border is desired on every page).

The footnote is pushed to the bottom of the page by virtue of the margin-top: auto style attribute and the fact that the <div> block having this attribute is within a vertical packed Flexbox container.

Notes:

• This was done with wkhtmltopdf 0.12.4, without the “wkhtmltopdf patches” according to the man page.
• If the height is too large on any page, all break-inside are ignored. In other words, the whole pdf document gets garbled, not just around the page that went wrong.
• I tried changing the resolution on my X11 display, and it didn’t make any difference. This might sound like a silly thing to check, but wkhtmltopdf depends on the X11 server.

Exactly like my previous post from 14 years ago, these are random jots that I took as I set up a QEMU/KVM-based virtual machine on my Linux Mint 19 computer. This time, the purpose was to prepare myself for moving a server from an OpenVZ container to KVM.

Other version details, for the record: libvirt version 4.0.0, QEMU version 2.11.1, Virtual Machine manager 1.5.1.

Installation

Install some relevant packages:

# apt install qemu-kvm qemu-utils libvirt-daemon-system libvirt-clients virt-manager virt-viewer ebtables ovmf

This clearly installed a few services: libvirt-bin, libvirtd, libvirt-guest, virtlogd, qemu-kvm, ebtables, and a couple of sockets: virtlockd.socket and virtlogd.socket with their attached services.

My regular username on the computer was added automatically to the “libvirt” group, however that doesn’t take effect until one logs out and and in again. Without belonging to this group, one gets the error message “Unable to connect to libvirt qemu:///system” when attempting to run the Virtual Machine Manager. Or in more detail: “libvirtError: Failed to connect socket to ‘/var/run/libvirt/libvirt-sock’: Permission denied”.

The lazy and temporary solution is to run the Virtual Machine Manager with “sg”. So instead of the usual command for starting the GUI tool (NOT as root):

$ virt-manager &

Use “sg” (or start a session with the “newgroup” command):

$ sg libvirt virt-manager &

This is necessary only until next time you log in to the console. I think. I didn’t get that far. Who logs out?

There’s also a command-line utility, virsh. For example, to list all running machines:

$ sudo virsh list

Or just “sudo virsh” for an interactive shell.

Note that without root permissions, the list is simply empty. This is really misleading.

General notes

• Virtual machines are called “domains” in several contexts (within virsh in particular).
• To get the mouse out of the graphical window, use Ctrl-Alt.
• For networking to work, some rules related to virbr0 are automatically added to the iptables firewall. If these are absent, go “systemctl restart libvirtd” (don’t do this with virtual machines running, of course).
• These iptables rules are important in particular for WAN connections. Apparently, these allow virbr0 to make DNS queries to the local machine (adding rules to INPUT and OUTPUT chains). In addition, the FORWARD rule allows forwarding anything to and from virbr0 (as long as the correct address mask is matched). Plus a whole lot off stuff around POSTROUTING. Quite disgusting, actually.
• There are two Ethernet interfaces related to KVM virtualization: vnet0 and virbr0 (typically). For sniffing, virbr0 is a better choice, as it’s the virtual machine’s own bridge to the system, so there is less noise. This is also the interface that has an IP address of its own.
• A vnetN pops up for each virtual machine that is running, virbr0 is there regardless.
• The configuration files are kept as fairly readable XML files in /etc/libvirt/qemu
• The images are typically held at /var/lib/libvirt/images, owned by root with 0600 permissions.
• The libvirtd service runs /usr/sbin/libvirtd as well as two processes of /usr/sbin/dnsmasq. When a virtual machine runs, it also runs an instance of qemu-system-x86_64 on its behalf.

Creating a new virtual machine

Start the Virtual Manager. The GUI is good enough for my purposes.

$ sg libvirt virt-manager &

• Click on the “Create new virtual machine” and choose “Local install media”. Set the other parameters as necessary.
• As for storage, choose “Select or create custom storage” and create a qcow2 volume in a convenient position on the disk (/var/lib/libvirt/images is hardly a good place for that, as it’s on the root partition).
• In the last step, choose “customize configuration before install”.
• Network selection: Virtual nework ‘default’: NAT.
• Change the NIC, Disk and Video to VirtIO as mentioned below.
• Click “Begin Installation”.

Do it with VirtIO

That is, use Linux’ paravirtualization drivers, rather than emulation of hardware.

To set up a machine’s settings, go View > Details.

This is lspci’s response with a default virtual machine:

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04)
00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 20)
00:04.0 Audio device: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller (rev 01)
00:05.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:05.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:05.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:05.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:06.0 Communication controller: Red Hat, Inc Virtio console
00:07.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon

Cute, but all interfaces are emulations of real hardware. In other words, this will run really slowly.

Testing link speed: On the host machine:

$ nc -l 1234 < /dev/null > /dev/null

And on the guest:

$ dd if=/dev/zero bs=128k count=4k | nc -q 0 10.1.1.3 1234
4096+0 records in
4096+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 3.74558 s, 143 MB/s

Quite impressive for hardware emulation, I must admit. But it can get better.

Things to change from the default settings:

• NIC: Choose “virtio” as device model, keep “Virtual network ‘default’” as NAT.
• Disk: On “Disk bus”, don’t use IDE, but rather “VirtIO” (it will appear as /dev/vda etc.).
• Video: Don’t use QXL, but Virtio (without 3D acceleration, it wasn’t supported on my machine). Actually, I’m not so sure about this one. For example, Ubuntu’s installation live boot gave me a black screen occasionally with Virtio.

Note that it’s possible to use a VNC server instead of “Display spice”.

After making these changes:

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Red Hat, Inc Virtio GPU (rev 01)
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
00:04.0 Audio device: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller (rev 01)
00:05.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:05.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:05.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:05.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:06.0 Communication controller: Red Hat, Inc Virtio console
00:07.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
00:08.0 SCSI storage controller: Red Hat, Inc Virtio block device

Try the speed test again?

$ dd if=/dev/zero bs=128k count=4k | nc -q 0 10.1.1.3 1234
4096+0 records in
4096+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 0.426422 s, 1.3 GB/s

Almost ten times faster.

Preparing a live Ubuntu ISO for ssh

$ sudo su
# apt install openssh-server
# passwd ubuntu

In the installation of the openssh-server, there’s a question of which configuration files to use. Choose the package maintainer’s version.

But with technical documentation (say, outlining an API) it can be very handy with something below \subsubsection{}. As it turns out, LaTeX actually supports lower levels, but they aren’t numbered by default. So it goes:

1. \section{}
2. \subsection{}
3. \subsubsection{}
4. \paragraph{}
5. \subparagraph{}

That’s neat, isn’t it? In order to make the two last numbered, add this to the LaTeX document:

\setcounter{secnumdepth}{5}
\setcounter{tocdepth}{5}
\titleformat{\paragraph}
{\normalfont\normalsize\bfseries}{\theparagraph}{1em}{}
\titlespacing*{\paragraph}
{-15ex}{3.25ex plus 1ex minus .2ex}{1.5ex plus .2ex}
\titleformat{\subparagraph}
{\normalfont\normalsize\bfseries}{\thesubparagraph}{1em}{}
\titlespacing*{\subparagraph}
{-12ex}{3.25ex plus 1ex minus .2ex}{1.5ex plus .2ex}

After adding this, sub-sub-sub-section numbers appear with \paragraph{}, and even one more level down with \subparagraph{}.

\label{} works as expected (\ref{} correctly references \paragraph{} and \subparagraph{}), and the table of contents also lists these elements neatly.

This snippet works well with the Hitec class. I don’t know if it works with other classes. But even if it does, odds are that the result will look ugly, as this code defines the spacing so that it looks fairly nice with Hitec’s formatting.

So it’s not really \subsubsubsection{}, which is awkwardly long anyhow, but a more elegant solution.

#define bits(x, y, z) (((x) >> (z)) & ((((long int) 2) << ((y) - (z))) - 1))

This picks the part that is equivalent to the expression x[y:z] in Verilog.

The cast to long int may need adjustment to the type of the variable that is manipulated.

And yes, it’s possible this could have been done with less parentheses. But with macros, I’m always compelled to avoid any ambiguity that I may not thing about right now.

What I really like about this pop-up is the “you’re missing out” part. I get the same thing from the silly image gallery app on my Google Pixel phone.  This is Google trying to play on my (not so existent) FOMO.

It has been suggested to add the –simulate-outdated-no-au argument to the command line that executes Chrome. This works indeed. The common suggestion is however to do that on the shortcut that executes the browser. But that won’t cover the case when I run the browser from a shell. Something I do, every now and then. Don’t ask.

So a more sledge hammer solution is to edit the wrapper script:

$ which google-chrome
/usr/bin/google-chrome

So edit this file (as root), and change the last line from

exec -a "$0" "$HERE/chrome" "$@"

to

exec -a "$0" "$HERE/chrome" --simulate-outdated-no-au='Tue, 31 Dec 2099' "$@"

What does this mean, then? Well, according to the list of Google Chrome switches, this switch “simulates that current version is outdated and auto-update is off”. The date is referred to in the source’s upgrade_detector_impl.cc. Look there if you want to figure out why this works (I didn’t bother, actually).

These post contains a few technical notes of using Google Translate for translating LaTeX documents into Chinese, Japanese and Korean. The insights on the language-related issues are written down in a separate post.

Text vs. HTML

Google’s cloud translator can be fed with either plain text or HTML, and it returns the same format. Plain text format is out of the question for anything but translating short sentences, as it becomes impossible to maintain the text’s formatting. So I went for the HTML interface.

The thing with HTML is that whitespaces can take different forms and shapes, and they are redundant in many situations. For example, a newline is often equivalent to a plain space, and neither make any difference between two paragraphs that are enclosed by <p> tags.

Google Translate takes this notion to the extreme, and typically removes all newlines from the original text. OK, that’s understandable. But it also adds and removes whitespaces where it had no business doing anything, in particular around meaningless segments that aren’t translated anyhow. This makes it quite challenging when feeding the results for further automatic processing.

Setting up a Google Cloud account

When creating a new Google Cloud account, there’s an automatic credit of $300 to spend for three months. So there’s plenty of room for much needed experimenting. Too see the status of the evaluation period, go to Billing > Cost Breakdown and wait a minute or so for the “Free trial status” strip to appear at the top of the page. There’s no problem with “activating full account” immediately. The free trial credits remain, but it also means that real billing occurs when the credits are consumed and/or the trial period is over.

First create a new Google cloud account and enable the Google Translate API.

I went for Basic v2 translation (and not Advanced, v3). Their pricing is the same, but v3 is not allowed with an API key, and I really wasn’t into setting up a service account and struggle with OAuth2. The main advantage with v3 is the possibility to train the machine to adapt to a specific language pattern, but as mentioned in that separate post, I’m hiding away anything but common English language patterns.

As for authentication, I went for API keys. I don’t need any personalized info, so that’s the simple way to go. To obtain the keys, go to main menu (hamburger icon) > APIs and services > Credentials and pick Create Credentials, and choose to create API keys. Copy the string and use it in the key=API_KEY parameters in POST requests. It’s possible to restrict the usage of this key in various ways (HTTP referrer, IP address etc.) but it wasn’t relevant in my case, because the script runs only on my computer.

The web interface for setting up cloud services is horribly slow, which is slightly ironic and a bit odd for a company like Google.

The translation script

I wrote a simple script for taking a piece of text in English and translating it into the language of choice:

#!/usr/bin/perl

use warnings;
use strict;
use LWP::UserAgent;
use JSON qw[ from_json ];

our $WASTEMONEY = 0; # Prompt before making request
my $MAXLEN = 500000;
my $chars_per_dollar = 50000; # $20 per million characters

our $APIkey = 'your API key here';

my ($outfile, $origfile, $lang) = @ARGV;

die("Usage: $0 outfile origfile langcode\n")
  unless (defined $origfile);

my $input = readfile($origfile);

askuser() unless ($WASTEMONEY);

my $len = length $input;

die("Cowardly refusing to translate $len characters\n")
  if ($len > $MAXLEN);

writefile($outfile, translate($input, $lang));

################## SUBROUTINES ##################

sub writefile {
  my ($fname, $data) = @_;

  open(my $out, ">", $fname)
    or die "Can't open \"$fname\" for write: $!\n";
  binmode($out, ":utf8");
  print $out $data;
  close $out;
}

sub readfile {
  my ($fname) = @_;

  local $/; # Slurp mode

  open(my $in, "<", $fname)
    or die "Can't open $fname for read: $!\n";

  my $input = <$in>;
  close $in;

  return $input;
}

sub askuser {
  my $len = length $input;
  my $cost = sprintf('$%.02f', $len / $chars_per_dollar);

  print "\n\n*** Approval to access Google Translate ***\n";
  print "$len bytes to $lang, $cost\n";
  print "Source file: $origfile\n";
  print "Proceed? [y/N] ";

  my $ans = <STDIN>;

  die("Aborted due to lack of consent to proceed\n")
    unless ($ans =~ /^y/i);
}

sub translate {
  my ($text, $lang) = @_;

  my $ua = LWP::UserAgent->new;
  my $url = 'https://translation.googleapis.com/language/translate/v2';

  my $res = $ua->post($url,
		      [
		       source => 'en',
		       target => $lang,
		       format => 'html', # Could be 'text'
		       key => $APIkey,
		       q => $text,
		      ]);

  die("Failed to access server: ". $res->status_line . "\n")
    unless ($res->is_success);

  my $data = $res->content;

  my $json = from_json($data, { utf8 => 1 } );

  my $translated;

  eval {
    my $d = $json->{data};
    die("Missing \"data\" entry\n") unless (defined $d);

    my $tr = $d->{translations};
    die("Missing \"translations\" entry\n")
      unless ((defined $tr) && (ref $tr eq 'ARRAY') &&
	     (ref $tr->[0] eq 'HASH'));

    $translated = $tr->[0]->{translatedText};

    die("No translated text\n")
      unless (defined $translated);
  };

  die("Malformed response from server: $@\n") if ($@);

  $translated =~ s/(<\/(?:p|h\d+)>)[ \t\n\r]*/"$1\n"/ge;

  return $translated;
}

The substitution at the end of the translate() function adds a newline after each closing tag for a paragraph or header (e.g. </p>, <h1> etc.) so that the HTML is more readable with a text editor. Otherwise it’s all in one single line.

Protecting your money

By obtaining an API key, you effectively give your computer permission to spend money. Which is fine as long as it works as intended, but a plain bug in a script that leads to an infinite loop or recursion, or maybe just feeding the system with a huge file by mistake, can end up with consequences that are well beyond the CPU fan spinning a bit.

So there are two protection mechanisms in the script itself:

• The script prompts for permission, stating how much it will cost (based upon $20 / million chars).
• It limits a single translation to 500k chars (to avoid a huge file from being processed accidentally).

Another safety mechanism is to set up budgets and budget alerts. Go to Main menu (hamburger) > Billing > Budgets & Alerts. Be sure to check “Email alerts to billing admins and users”. If I got it right, budgets don’t protect against spending, but only sends notifications. So I selected a sum, and enabled only the 100% threshold. It seems to make sense to check all the Discounts and Promotion options in the Credits part, which makes sure that the alert is given for the money to be spent by deducing all promotion credits.

On top of that, it’s a good idea to set quota limits: Go to Main menu (hamburger) > IAM & Admin > Quotas. Set the filter to Translation to get rid of a lot of lines.

It’s also the place to get an accurate figure for the current consumption.

Enable the quota for “v2 and v3 general model characters per day”, which is the only character limit that isn’t per minute, and set it to something sensible, for example 2 million characters if you’re a modest user like myself. That’s $40, which is fairly acceptable damage if the computer goes crazy, and high enough not to hit the roof normally.

Also do something with “v3 batch translation characters using general models per day” and same with AutoML custom models. I don’t use these, so I set both to zero. Just to be safe.

There’s “Edit Quotas” to the top right. Which didn’t work, probably because I did this during the trial period, so quotas are meaningless, and apparently disabled anyhow (or more precisely, enabled to fixed limits).

So the way to do it was somewhat tricky (as it’s probably pointless): To enable a quota, right-click the “Cloud Translation API” to the left of the quota item, and open it in a new tab. Set up the quota figure there. But this description on how to do it might not be accurate for a real-life use. Actually, the system ignored my attempts to impose limits. They appeared on the page for editing them, but not on the main page.

Supporting CJK in LaTeX

I’m wrapping up this post with notes on how to feed LaTeX (pdflatex, more precisely) with Chinese, Japanese and Korean, with UTF-8 encoding, and get a hopefully reasonable result.

So first grab a few packages:

# apt install texlive-lang-european
# apt install texlive-lang-chinese
# apt install texlive-lang-korean
# apt install texlive-cjk-all

Actually, texlive-lang-european isn’t related, but as its name implies, it’s useful for European languages.

I first attempted with

\usepackage[UTF8]{ctex}

but pdflatex failed miserably with an error saying that the fontset ‘fandol’ is unavailable in current mode, whatever that means. After trying a few options back and forth, I eventually went for the rather hacky solution of using CJKutf8. The problem is that CJK chars are allowed only within

\begin{CJK}{UTF8}{gbsn}

[ ... ]

\end{CJK}

but I want it on the whole document, and I need the language setting to be made in a file that is included by the main LaTeX file (a different included file for each language). So I went for this simple hack:

\AtBeginDocument{\begin{CJK}{UTF8}{gbsn}}
\AtEndDocument{\end{CJK}}

As for the font, it appears like gbsn or gkai fonts should be used with Simplified Chinese, and bsmi or bkai for with Traditional Chinese. Since I translated into Simplified Chinese, some characters just vanished from the output document when trying bsmi and bkai. The back-translation to English of a document made with bsmi was significantly worse, so these dropped characters had a clear impact in intelligibility of the Chinese text.

I got this LaTeX warning saying

LaTeX Font Warning: Some font shapes were not available, defaults substituted.

no matter which of these fonts I chose, so it doesn’t mean much.

So the choice is between gbsn or gkai, but which one? To decide, I copy-pasted Chinese text from updated Chinese websites, and compared the outcome of LaTeX, based upon the TeX file shown below. It was quite clear that gbsn is closer to the fonts in use in these sites, even though I suspect it’s a bit of a Times New Roman: The fonts used on the web have less serifs than gbsn. So gbsn it is, even though it would have been nicer with a font with less serifs.

For Japanese, there’s “min”, “maru” and “goth” fonts. “Min” is a serif font, giving it a traditional look (calligraphy style) and judging from Japanese websites, it appears to be used primarily for logos and formal text (the welcoming words of a university’s president, for example).

“Maru” and “goth” are based upon simple lines, similar to plain text in Japanese websites. The latter is a bit of a bold version of “maru”, but it’s what seems to be popular. So I went with “goth”, which has a clean and simple appearance, similar to the vast majority of Japanese websites, even though the bold of “goth” can get a bit messy with densely drawn characters. It’s just that “maru” looks a bit thin compared to what is commonly preferred.

Korean has two fonts in theory, “mj” and “gt”. “mj” is a serif font with an old fashioned look, and “gt” is once again the plain, gothic version. I first failed to use the “gt” font even though it was clearly installed (there were a lot of files in the same directories as where the “mj” files were installed, only with “gt”). Nevertheless, trying the “gt” font instead of “mj” failed with

LaTeX Font Warning: Font shape `C70/gt/m/it' undefined
(Font)              using `C70/song/m/n' instead on input line 8.

! Undefined control sequence.
try@size@range ...extract@rangefontinfo font@info
                                                  <-*>@nil <@nnil

But as it turns out, it should be referred to as “nanumgt”, e.g.

\begin{CJK}{UTF8}{nanumgt}
나는 멋진 글꼴을 원한다
\end{CJK}

It’s worth mentioning XeLaTeX, which allows using an arbitrary True Type font withing LaTeX, so the font selection is less limited.

See this page on fonts in Japanese and Korean.

For these tests, I used the following LaTeX file for use with e.g.

$ pdflatex test.tex

\documentclass{hitec}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{CJKutf8}
\newcommand{\thetext}
{

它说什么并不重要，重要的是它是如何写的。
}

\AtBeginDocument{}
\AtEndDocument{}
\title{This document}
\begin{document}

gbsn:

\begin{CJK}{UTF8}{gbsn}
\thetext
\end{CJK}

gkai:

\begin{CJK}{UTF8}{gkai}
\thetext
\end{CJK}

bsmi:

\begin{CJK}{UTF8}{bsmi}
\thetext
\end{CJK}

bkai:

\begin{CJK}{UTF8}{bkai}
\thetext
\end{CJK}

\end{document}

This post summarizes my insights as I worked my way through translating some technical documents, written in LaTeX, into Chinese, Japanese and Korean. The immediate approach was to feed Google Translate with the pdf documents, but not only are the results ugly, but then there are a lot of technical terms in the documents which are better not translated. Even worse, there are code examples with explanation in the text, file names, references to variable names and other elements that become meaningless if translated.

One of the advantages of having the document written in LaTeX to begin with, is that the LaTeX text formatting commands effectively flag the parts that aren’t just plain language in the text, so it’s relatively easy to spot them and protect them. But that alone was a long way from the finish line, as elaborated in this unexpectedly long post.

A different post discusses the technical aspects of talking with Google Cloud’s API as well as creating documents in these languages with LaTeX.

I also did something similar with translating web pages. For example, the translation of this post to Chinese, Japanese and Korean.

This post was written in the summer of 2022, and odds are that things will change dramatically over the course of time.

Is translation by human better?

The short answer: Yes, as of 2023, human translation is much better. It’s mainly because there is no way to give the translating tool hints about the context. For example, the word “driver” could be a car driver or a term related to a computer. All translation tools just pick one meaning. Some tools allow choosing a specific dictionary, and there are ways to shape the behavior of the translator. But the results are far from satisfactory.

However but both options have their disadvantages: Working with a human necessarily requires trusting that a specific person will perform the job thoroughly, and well, that’s anything but taken for granted. It’s extremely difficult to verify that the work was done well, in particular when the document is technical, as it’s not possible to give it to just someone and ask if it’s well written. An automatic reverse translation will miss some poor translations (in particular poor translations of technical terms) and at the same time make false alarms.

But the worst problem with human translation is that every future change in text requires contacting the people who made the translation, and ask them to make the adjustments. They may not be so willing to do that. So unless you employ these people full-time, it may be difficult to translate small edits.

Another problem with humans is that significant errors in the meaning of the text might occur. It’s easy to reverse or otherwise obscure the meaning of a sentence because of a simple human error. “Be sure not to turn on the power supply” can easily turn into “Be sure to turn on the power supply”. Automatic reverse translation can reveal this, but it’s easy to miss an error like this, when the person that verifies the text already knows what it should say.

Automatic translation should be less likely to make a mistake of this sort, but the truth is that Google Translate, with all its Neural Network magic, turns out to be more human than desired in this matter: It’s not completely unusual that the meaning of the text changes, sometimes to the complete opposite.

It also has a variety of passive-aggressive behaviors, in particular ignoring whole sentences or part of them, mostly when the text becomes a bit rambling.

I had a case where the automatic translation ignored a “non-” prefix on a noun, and by doing so reversed the meaning of the sentence. I’ve also had a case where “must not” was translated into the equivalent of “doesn’t have to”.

The clear disadvantage of an automatic translation is poor expression and grammar. If the technique explained below is adopted, it’s however possible to end up with a fairly good result, even if the language is a bit off at times.

But this disadvantage can be mitigated by letting someone who knows the target language well proof-read the result. This person doesn’t need to know English well, but only be sensitive to the target language, so it’s easier to find someone for that job. And in particular when translating to Asian languages, it’s easy to tell the persons involved to ignore technical terms, as they are easily distinguishable, written in Latin script.

The results of this proof-reading session are only slight changes in word choice or ordering, and they can be verified against automatic translation as well as another person. In fact, in most cases, the best way is to improve the wording in the original language, until the person checking the text confirms it sounds OK.

Whether it’s worth the effort and cost to make this language cleanup is an open question. It’s a matter of how much the target audience appreciates the fact that the documentation is available in their language vs. how much the language defects come across badly.

Another issue with automatic translation is that words with more than one meaning can be mistranslated, in particular when the intended meaning is the less common one for a specific word (examples for that below). A back-translation doesn’t necessarily reveal a problem of this sort.

So with the possibility of having someone read through the translated text, the only remaining problem is when the meaning is changed unnoticed during the translation. Frankly speaking, I don’t know which option, human or machine, is better regarding this problem. The only real solution anyhow is to back-translate the text and read it through. Good luck with that.

General insights on automatic translation

Google Translate is based upon a neural network machine learning algorithm, which means that it’s chaotic by nature (in the scientific sense). That gives it a bit of a human touch, which surely makes the translations better, but also makes it quite unpredictable. In particular, it’s capable of making any possible mistake, no matter how pointless and unexpected. It’s impossible to be 100% sure that it won’t do this or that, and it’s not even a bug when a phrase in the original text just disappears, or when a meaningless string of characters gets translated to something else, also completely meaningless. Those small glitches are part of the game, and it makes automated processing of the translated text quite challenging.

Having said that, the general rule is that if Google Translate does weird things, it’s because it was fed with something it found hard to digest. So even if the weirdness doesn’t appear to be related to language, the best way to rectify this is to change the original text into a simpler, more common way to express the same idea. Unfortunately, this calls for dull, play-it-safe English. However with by far less silly typos and grammar mistakes.

If I was to speculate how Google Translate’s algorithm works, I would say something like this: Attempt to find recognizable words in the sentence, fix spelling mistakes (“did-you-mean” style) and try to match the words that are recognized with a known phrase from the huge training corpus. Pick the known translation into the desired language of the word pattern that fits best. Fill in the words that were unknown in the original language in the translated text in their natural positions — these are treated as names (of persons, places etc.).

Punctuation like full period and commas, as well as enclosure in parentheses, makes the translator treat each part separately, more or less.

The destination language matters a lot regarding the interpretation of the meaning of the text. It doesn’t seem like the question is “what does the original text mean” but “which language pattern was most common in the training data for translating into language X”.

The main takeaways for this speculation, regarding how to write for translation are:

• Use common expressions and language patterns. In particular, use the most commonly used words to express a certain meaning.
• Be super-careful with trivial spelling mistakes, as they break the statistics for the desired language pattern.
• If the translation is successful to one language, in the sense that the original meaning was “understood”, it doesn’t necessarily means it will be as successful to another one. Same goes with failure. It seems to depend on what the translations between the two languages are usually used for. In other words, judging by the results, it seems like translations into Hebrew are used more for everyday text, but translation into east Asian languages is more for technical documents. Hence the selection of meaning tends to be more technical with the latter.
• As there is no analysis of the semantics of the original sentence, anything can happen, including a translation that says the opposite of the original.

Interestingly enough, I’m under the impression that the translation with Google Lens is much better than the cloud based translation service. In particular, the cloud translation is more likely to produce nonsense translations because of small disturbances in the middle of text, where Google Lens’ translation seems to have extra wisdom to overcome such.

Translating to a foreign language

How do you know a translation is OK, when you don’t know the language it’s translated into? The short answer is that one can’t really know. It helps a lot knowing another language, even if it’s different from the target language, because it allows spotting misinterpretations of certain words, in particular technical ones. But often a word is poorly translated into one language, but fine with another.

There’s the possibility to translate it back to English, but that doesn’t always spot problems. Technical words like “clock”, “bus”, “sink”, “assertion” are translated to ridiculous words in Hebrew, for example, but the translation back looks OK in English. In particular a work like “sink” translates into the word in Hebrew that means kitchen sink, and then goes back to the correct work in English, of course.

But then comes the question: Why translate these words at all?

Quality of translation

Among the three target languages, the translation to (simplified) Chinese is the best by far. Probably because the natural flow of Chinese is the closest to western languages. The runner-up is Korean, and the worst is Japanese.

The worst problem with both Korean and Japanese is that parts of the original text can just disappear. This happens often when the semantic structure gets too complicated, or if there’s no normal way to say something in Japanese. For example, the sentence “you’re absolutely welcome to mess up completely, the tools won’t stand in your way” lost the entire first part in Japanese. So it just says “no tools get in the way”. If only the first part is translated separately, it turns into “completely ruined is welcome” (it had to give me something back when that sentence stood alone).

So short and plainly informative sentences are best translated into Japanese and Korean. Chinese seems to work with anything.

As for words like “it”, Chinese tolerates that best too. The two other languages are more likely to need repeating the word that “it” refers to, and hence possibly pick the wrong word to repeat.

Testing by translating to Hebrew

Since I happen to speak Hebrew fluently, I checked the translation to Hebrew of all documents, not for the purpose of publishing this translation, but because I soon found out that Google Translate struggles with Hebrew. So the idea was that if it’s OK with Hebrew, it’s probably OK any language.

For this, I tried two cases where the translation got wrong, as indicated by the result in Hebrew.

The first sentence that failed was “Close the window after a successful generation”. The problem was that the word “generation” was interpreted as the relationship between age groups, and not from the word “generate” as intended. This, in itself, is easily fixed by changing it into “Close the window after a successful generation of the file“. It was a matter of fitting the entire sentence into a different pattern of words.

Surprisingly enough, the translation into Chinese, Japanese and Korean was correct even without the fix. This can be verified by looking at the translation back to English, and isolate the word or couple of words of interest.

The next problematic phrase was “The non-X81X are ignored by almost all X82X computers”. In the translation to Hebrew, the “non-” part was ignored, so the sentence’ meaning was reversed. Once again, the translation into the three other languages was correct (the X81X thing is explained below).

So if I once had the speculation that the machine translates the words into an intermediate format that somehow contains the meaning, and takes it into the target language from there, it’s definitely not the case. Whether there’s a misunderstanding or not in the translation depends on the target language.

I’m optimistic and hope that Hebrew is in particular prone to errors, so if I clean up the translation to Hebrew, it will hopefully work with other languages. However odds are that each language has its own pitfalls. Even though it really seems like the translation to Hebrew from any language is bad in particular. Including changing the meaning of the text. Also, I’ve found that plain idioms like “it doesn’t hurt” are often translated horribly to Hebrew but get perfectly OK in CJK languages. But then, I don’t know about misses in CJK languages that were OK in Hebrew…? And yet, after checking numerous expressions (“bite back”, “copy-paste” and a lot of this sort) it really seems like Hebrew is really bad off.

This way or another, the only sure benefit of checking the translation to Hebrew is that it does, after all, remove some ambiguities, whether that is necessary or not. Actually, I found tons of plain typos by looking at this translation, so that alone justifies this phase. It’s difficult to proofread text exactly as it was written, but reading it again in another language feels as if someone else wrote it.

I also had the opportunity to have a translation into Japanese by a helpful person, and it was quite clear that the problems were in the places where the Hebrew translation also limped.

Hands-on insights

After quite some back and forth, I learned that the best way to work with Google Translate with text is to feed it with paragraphs of text in HTML, enclosed in <p> (or <hN>) tags. Plain formatting tags is fine (<b>, <i> and even <span> etc.) but it’s important not to insert anything that breaks the continuity of the sentences: No <br> or <img> tags in the middle, or anything else that isn’t expected in the middle of a sentence. It makes Google Translate translate the part before and after the break as separate sentences, and that’s a disaster.

Newlines are ignored in the cloud interface with HTML, as they should be. This is contrary to the web interface for Google Translate, which is extremely sensitive to newlines, so copy-pasting a chunk of text from a pdf document can result in a horrible translation, because there are newlines between each row in the original text, which makes the translator treat each row a separate phrase.

But the real difficulty is the fact that the translated text is technical. Google Translate is trained with mainly non-technical text (I guess), so its interpretation of technical terms that happen to also have a non-technical meaning is naturally inclined towards the non-technical meaning. Words like “driver”, “compile”, “assert” and “board” are not only likely to be translated incorrectly, but also stir a mess in that imaginary brain that holds all those neurons, resulting in a completely unintelligible translation.

The problematic words are those that have a possible non-technical meaning. The word “boot” could mean a piece of footwear, to boot a computer could be mistaken for “to give the computer the boot”, but to reboot a computer could only mean one thing. So it’s not all that much about the word being technical, like the fact that it could be remotely confusing.

Other ambiguities occur with words like “target”. Using it in any form, i.e. “to target” or “targeting” as well as “targeted” as in “depending on which software version is targeted” leads to a completely wonky translation, at least into Hebrew.

Surprisingly enough, it copes quite well with sentences that contain untranslatable items. I guess it treats anything it can’t handle as a name. Since it’s supposed to be able to translate “Joseph prefers Paris over Berlin”, it works fine with “X prefers Y over Z” as well. So the trick is to remove all technical terms from the the text, and replace them with something that Google Translate will treat as a name, something it can’t translate. And then return those words into the translated text.

This means that all technical terms remain in English in the translated text, which is perfectly fine, because a technical reader is expected to know these terms. It’s the blah-blah part that needs translation, and with the technical words out of the way, Google Translate does a good job on that.

The problem that remains is how to feed the translator with these untranslatable X, Y and Z placeholders, when there can be thousands of these, and they must all be left intact in the translation (well, except for Russian and Greek, see below). The section below on placeholders tells the full story, but the spoiler is that I used X0X, X1X, X2X, and so on. It’s not watertight, but it works best. I tried quite a few options.

The main thing to keep in mind is that it’s all about word patterns: If Google Translate recognizes the structure of the sentence, based upon words that are commonly used together for a certain meaning, it translates that part correctly, and then puts the placeholders in the right places, treating them as names.

I should mention that Google Translate offers a “notranslate” style, which can be used to enclose e.g. <span> segments of text that shouldn’t be translated. I didn’t attempt using it, in particular as people out there in the web have complained that it disrupts the translation exactly like that. Another problem is that chunks that shouldn’t be translated often have a different formatting (e.g. Courier font for variable names), and Google Translate tends to behave in an unpredictable manner, making it difficult to rely on its output for feeding LaTeX with directly.

Also worth mentioning is that Google offers an advanced version of the translation API, with the ability to train the learning machine and possibly feed it with specific word translations, but that would require knowing the correct term in the target language. How do you say “compile” in Chinese and Japanese? But it could have been helpful for solving the problem with verbs, that have a technical meaning (“compile”, “boot”, “implement”, “overflow”, you name it).

How I actually did it

The idea was to extract translatable text from the LaTeX source, and feed Google Translate’s cloud API with it in HTML mode. Then take the translated text and implant it back into the LaTeX doc.

The overall goal is to feed Google Translate with hollow phrases, albeit with a solid and common semantic structure, of the form of “X with Y is more important that Z”. This makes it easy for the translator to detect the structure of the phrase, and translate it to a meaningful sentence in the foreign language. That gives good odds for meaningful sentence when the placeholders are replaced with the actual technical words in the translated phrase.

In more detail:

• Fetch paragraphs of text and enclose them in <p> or <h1>, <h2> or <h3> tags. Each of these tags have a unique “id” attribute, so when the translation returns, it’s possible to track which text segments should be written back to which part in the LaTeX file. This is why HTML mode came handy. I haven’t had a single case of these attributes being messed up (yet?).
• Turn some LaTeX formatting into plain HTML tags, e.g. <b>, <i> etc. Then do the opposite when implanting the text back. The advantage is that this doesn’t break Google Translate’s view of the text as a contiguous sentence. Once again, HTML mode is required for this stunt.
• Anything that shouldn’t be translated — technical terms, references to variables, file names, references to paragraphs, labels etc. — is replaced with a unique identifier (“placeholder”) that Google Translate doesn’t attempt to translate. The major caveat with this method is that it works only with nouns. This requires rewording, in particular turning verbs into nouns (e.g. “perform a compilation” instead of “compile”). More on this below.

Note that some parts of the LaTeX document are completely out of this game, as they aren’t even given to the translator to look at. For example, verbatim environment chunks, and even the newlines between the text paragraphs. They remain the same because they aren’t overwritten when the translated text is transformed back and implanted in the relevant segment.

Work flow

I wrote a Perl script for the back-and-forth manipulations between LaTeX and HTML, but I won’t get into that too much, because it’s complicated and really specific to the documents at hand for translation. Among others, this script loaded a list of words that are always replaced with placeholders, and I also added a LaTeX command, \notranslate{}, which just leaves the content as is when interpreted by LaTeX, but to the script it means that the entire chunk should be replaced with a placeholder as well.

Writing scripts and all that is nice, but there’s still some manual preparation required. So this was the procedure I adopted:

• Run the script that creates the HTML file that is sent to Google Translate. View that file with a web browser, and look for words that are technical and can be mistranslated. When such are found, either add the word or phrase to the list of words to automatically replace with placeholders, or fix it specifically with \notranslate{} LaTeX statements.
• In fact, I also wrote a script that puts \notranslate{} on certain words and patterns (e.g. sets of upper case characters) so I ran this script, and then verified each such occurrence. This is faster than finding them manually, and is useful for words that may have a non-technical meaning, or otherwise require human attention to get 100% right. For example, the word “image” should be translated when it refers to a picture in the text, but not when it’s an image of a disk.
• Go through the text manually, and apply the guidelines listed below (the do’s and don’ts).
• Translate the text into Hebrew, and read through the result. If something ends up unclear, fix it. The further the language is from English, the better. The one sure benefit of this check is that small typos are spotted (e.g. “in” instead of “is”) because the translation gets weird. The fact that the order of words changes in the translation also helps spotting ambiguities, that are often solved with works like “which is” or punctuation.
• Translate into the target language. Make the necessary fixes. Don’t bother to find out why certain placeholders are missing in the translation. Rather, look at the original text and try to figure out why it was difficult to translate, and fix that instead. Sometimes a missing placeholder is due to a whole sentence being dropped off, in particular with Korean. It’s as if the algorithm said “I have no idea how to reorganize this sentence into something that makes sense in Korean, so I’ll just skip it”.
• Maybe attempt to translate the document back as a pdf file (with Google Translate’s web interface) or use Google Lens’ translation feature for a more sporadic check. I’m not sure if this is worth the time.

The order of translation is Korean first, then Japanese and finally Chinese. This is because the translation to Korean is the most troublesome, however often fixing the problems consists of changes that are likely to benefit the other translations.

All in all, it appears like using placeholders instead of technical terms actually improved the translation regardless of these specific words. It seems like these words confused the translation machinery, which made it create obscure phrasing. With the technical words out of the way, inserted as opaque symbols, it seems like Google Translate managed much better to handle the rest, which now consisted of commonly spoken language.

So my ultimate approach was to put placeholders instead of virtually all technical terms which are nouns. That’s quite a lot of them, and the translated documents ended up full with terms in English. I’m not sure what Chinese are going to think about this, but if they have the same problem as in Hebrew — weird “official words” for technical terms — it’s going to be just fine.

The do’s and don’ts

Based upon quite some trial and error, these are the guidelines I ended up with for producing text with placeholders that translates well.

• The text should consist of hollow sentences like “If the X113X process fails, the X641X’s output may supply hints on what went wrong. The X102X application on the computer should be configured for X114X, no X115X and no X116X ( X640X )”. However sentences like “The X219X for X220X on X221X or X222X is part of the majority of X223X and distributions, as explained below” should be fixed, inserting some meaningful words between those four placeholders with just one word between each. In this example, it’s not clear whether the last “or” refers to instead of X221X alone or all of the three. If the translation requires word reordering, this will obscure the meaning of the sentence.
• Use punctuation (in particular commas) and parentheses to chop up long sentences into segments. This prevents ambiguity. In particular, text inside parentheses is translated into parentheses, so this is a good tool for breaking up long and complicated sentences.
• Try to keep phrases short and concise (and somewhat boring), partly because sentences are short in the target languages. If the sentence is long, try to mitigate the damage with punctuation.
• Use plain and explicit English. Don’t leave out “which”, “that” and all those words that explicitly define the role of each word. Even a simple expression like “for the curious” can go wrong, but works perfectly well when changed into “for those who are curious”. Yuck, but translates well.
• Avoid words that refer back to something earlier in the sentence, unless it’s very obvious. In particular, the word “it” is often replaced with the word it’s supposed to refer to during the translation, and sometimes the wrong word is chosen in the translation. When this happens, the translation explicitly changes the meaning. Because the translation into CJK languages often involves splitting a long sentence into shorter ones, without a possibility to use a word like “it”, implicit references of this sort easily translate into nonsense. To make things worse, the back-translation may bring back the “it”, so there’s no way to spot the mistaken translation. There are cases where these duplications are safe, for example expressions like “one thing to another” (which is often translated into “one thing to another thing”).
• Prefer “the red book and the blue book” over “the red and blue books”. The order of the words may be changed during the translation, and in that case, it’s possible that only the “blue books” is moved to the new position in the sentence, so the result is rubbish. These overly explicit sentence are less awkward to read than they are awkward to write, but they are nevertheless slightly awkward as the same word is repeated over and over again.
• Avoid idioms. Even the simplest ones, like “out of the box” may and may not translate into something that makes sense. Because of the statistical nature of the translations, idioms might get translated with the right spirit into a certain language, and fail completely with another. So dull language it is.
• Avoid verbs in passive form, in particular if it comes with a “by”. Passive form is useful for not naming the doer, but if it’s named anyhow, use the active form. A lot of times, the passive form, and the tangled sentences that it usually creates, were the reason for problems in translation.
• Use possessive form for clarification. For example, if the word “register” is replaced with a placeholder, “register modification” should change to “modification of registers” or likewise “registers’ modification”. Using the ‘s suffix works great, so use it as long as it doesn’t create an ambiguity on who the owner is.
• In fact, there’s no problem at all with segments like “X400X’s X401X”, as possessive form. This translates well, surprisingly enough.
• Don’t replace partial expressions with placeholders. For example, in the expression “the user-space application”, don’t replace just “user-space”, but rather “user-space application”. Word ordering might be different in another language, which can at worst lead to a complete disassociation between the placeholder and its related word in English, with a completely unclear result.
• Avoid replacement of parts of expressions with placeholders. For example, in “VGA port”, if only “VGA” is replaced, it’s not sure if this will translate fine. “VGA-port” increases the changes. If it’s a common language pattern, e.g. “VGA-based”, there’s a good chance for proper translation. Same goes with “the X500X file”, because it’s a common language pattern.
• Don’t use “non-” as a prefix. It’s often missed, reversing the meaning.
• Look out for ambiguous words. For example, the word “signals” could be the verb (to signal) but also the plural of the noun. Avoid less common uses of words, such as “writes” to say several write operations, and use “write operations” instead.
• Be extra careful with trivial spelling mistakes and typos, in particular mixing “is” with “it” and such. These are overlooked when reading the text in English, but they kill the translation, sometimes by changing the meaning significantly, and sometimes by just confusing the translation algorithm into producing something weird.
• Bonus: Check all duplication of placeholders, and verify that the correct one is duplicated. Because these duplications are usually the result of a word that refers back to something (“which”, “that”, “it” etc.), it’s a good idea to verify that the reference goes to the correct placeholder. In theory, this should be done with all uses of back referencing, but that means proofreading the entire text. So with placeholders it’s less work (and less gain). Having run through a checkup of my own translations, I’d say about 10% of these duplications garble the meaning, by explicitly duplicating the wrong word.

Caching translation results?

Since the document is chopped into paragraphs, each within a <p> enclosure, does it matter if each is sent separately or if all are sent in one API transaction as a concatenated string? Does it matter if the translator sees the entire text?

Because if each <p> enclosure is treated separately, it’s possible to cache the pieces of text that have already been translated.

Caching is more than just a money saver. It allows making manual changes in Google Translate’s output (in particular if it messed up the placeholders) and then not having to repeat this every time the entire document is translated.

Even more important, avoiding the repeated translation of parts that have already been translated means avoiding the possible mishaps that may suddenly occur (like suddenly dropping a sentence). Think about making a small change, and then the translation fails on something completely different. But it worked last time!

This is also important if there’s feedback from readers that corrects a poor translation at a specific place. So caching is very helpful for the incremental kind of work that is necessary to maintain the document in the long run.

So I tried this with translating from English to Hebrew, and a bit with Chinese as well (by looking at the translation back to English). As it turns out, there are occasional differences between the translation of an isolated paragraph and that made with a context. But it doesn’t seem like an intelligent use of the context. Comparing the results, the isolated translation was sometimes better, sometimes worse, with a very slight difference in most cases. So it looks more like the algorithm randomly picked another wording, for no apparent reason. It was usually exchanging equally valid synonyms, or choosing to translate the name “Linux” to Hebrew or not.

Another observation I made is that the use of context is poor. For example, the word “call” is translated to the the word in Hebrew that means a phone call, but “function call” is translated correctly. So what if there’s a sentence saying something about a “function call”, and a sentence afterwards uses the word “caller”? In the <p> enclosure, that is. Well, the translation of “caller” still relates to a phone call. The neural network clearly didn’t learn anything from the first sentence.

So it makes perfect sense to cache translations at a paragraph level. If the original document changes, request a translation only on the enclosure that actually changed.

Finding the right kind of placeholder

This is a long explanation on why ended up with the XnX placeholders. I would skip this part if I were you.

As mentioned above, the main problem with translating a technical document is that some technical terms are translated into an unhelpful, sometimes ridiculous way, and that it confuses the translation algorithm. As the reader of the document is most likely familiar with the English term, it’s safer to leave these words as is. The problem is how to insert these terms in a way that ensures they don’t get translated, and at the same time retain their position in the context.

As it turned out, the main problem with inserting an untranslated chunk into the text is that it may disrupt the translation, in particular as Google Translate tends to treat the part before and after the chunk as separate sentences, which results in a poor translation that misses the point of the sentence.

I began with adding markers in a plain text (like <%103%>, [^29^] and ^^26^^), however Google Translate inserted a space in the middle of some of these (so it turned out to be e.g. “< %103%>”) and also threw in some markups where they shouldn’t be. A complete disaster, in short. This could have worked with non-HTML translation, but well, it didn’t work.

Another attempt was to use translation of HTML, with <b id=”n23″>P</b> markers as placeholders. The id allowed to identify which placeholder to insert, and the “P” to give the translator something to consider as a word. This failed as well, in many ways: The fact that the “P” part sometimes got translated into “PP” (why on earth) didn’t matter much, because it’s not really important. The real problem was that at times there were other words inserted into the <b> enclosure as well (for no apparent reason). Even worse, sometimes a completely different word, somewhere else in the sentence, got into a <b> enclosure with the same id. So processing this would have been complicated.

Another thing I tried was to use <var>n</var> enclosures, where n is the number of the placeholder. That failed, partly because some of these disappeared for no clear reason, and others were manipulated (for example, characters from previously outside the enclosure went into it).

To ensure that the placeholder is fully opaque, I tried <img id=n23>. The clear advantage was that Google Translate didn’t duplicate these not modify them, but they broke the sentence into fragments. Google Translate assumed that no sentence will have an image in the middle of it.

So if not images, what about emoticons? Or even better, I made an attempt to use the Unicode range U+13000 to U+1342e (Egyptian Hieroglyphs) as placeholders instead of <img> markups. The idea was that Google Translate would have to pass them through as is, and that they would be considered to be names. In order to make this work, there had to be a whitespace on both sides of the Hieroglyph, but even with that, Google Translate would mess up and occasionally add completely unrelated characters instead.

In the end, I went for inserting words like X0X, X1X, X2X, and so forth. These remain intact through translation, however they are occasionally duplicated, in particular with sentences like “that is possible with X, which is the best option” which can turn into “that is possible with X, and X is the best option”. The word “it” is also translated sometimes into the placeholder instead. But that’s actually a correct translation, and it’s easy to process. Even though this worked almost flawlessly, there were occasional surprises, including rare cases where Google Translate changed the number between the Xs without myself being able to figure out why on earth, and why that specific change. So there’s always a certain amount of manual cleanup after the translation.

These duplications are common with east Asian languages, and usually occur when a long sentence is chopped into several shorter ones. In these languages, it’s more common to repeat the word than to use “it”, “which” and such.

When translating to Russian and Greek, the “X” character was occasionally replaced with the Russian capital letter “Ha” (Unicode U+0425) or the Greek capital letter “Chi” (Unicode U+03A7). Both look exactly like an “X”, so the replacement is understandable. Once this issue is known, it’s quite easy to handle, so it’s not a big deal.

As for the quality of the translation, this worked well, and Google Translate combined these nicely into the translation, even when changing the word ordering was necessary. This works however only when the placeholder is used as a noun. So it doesn’t solve the problem with verbs like “assert”, “raise”. In some cases, a word like “overflow”, used as a verb, can be replaced with something like “cause an overflow”, so it can be translated properly.

Another thing with these XnX placeholders is that there must be a whitespace in either side of it, or Google Translate gets confused. To ensure that the placeholder is restored properly, the strategy was to include any surrounding whitespaces in the string that was stored to replace the placeholder later on, and then add a whitespace in either side of the XnX string. When reverting the process, all whitespaces around the XnX string were removed before restoring the original string. This results in a perfectly consistent back-and-forth, even if the translator adds or removes whitespaces (which happens a lot).

As a side note, Google charges for all characters, even those not translated. Hence it’s a good idea to keep the placeholders short markups. Not a big deal, but still.

Sanity checks on placeholders

The natural expectation is that any placeholder in the text for translation will result in a single placeholder in the translation. I’ve already mentioned above that some placeholders turned into two in the translated text, and it was actually correct. But what if the placeholder disappears?

The answer is that it’s always an error, and it has to be fixed manually. In fact, it’s often an indication that something worse happened, which would have been left unspotted had it not been for the missing placeholder. Sometimes the number between the Xs is changed arbitrarily, but it happens in conjunction with other placeholders in the vicinity being messed up.

Sometimes the absent placeholder was the result of a part of a sentence that was completely eliminated. The small piece of information it contained was simply absent in the translation. This can happen for several reasons, but the most recurring one seems to be when it’s not clear what “which” or “that” refers to, earlier in the same sentence. One can get away with that in translations to European languages, but because the sentence is built differently in east Asian languages, the translator is forced to make a pick. So instead of doing that, it just eliminates the part it can’t decide upon. A neural network algorithm showing a bit of human behavior, I would say.

It also seems that a colon sign (‘:’) tends to eliminate what comes immediately after it, fully or partly. Changing it to a full stop often returned chunks of texts from the dead in Korean and Japanese. Or splitting the text, so that part after the colon is in a separate enclosure (note to self: possibly with a \skipthis{}).

Same thing with a sentence starting with “likewise”.

Another somewhat weird phenomenon with Korean and Japanese is that a whole sentence was sometimes dropped. The really weird thing was that when the same sentence was put in a separate <p> enclosure, it was translated properly. So it was like Google Translate said “nah, this is too much rubbish, I’ll drop the last sentence”.

So in this sense, the placeholders help spotting other problems with the translation. I got an error of this sort for each few thousand translated words, which practically means a bit of fixing for each document. What’s really worrying is how many sentences without any placeholders have vanished unnoticed?

Placeholders that contain a word in plural

One problem that is inevitable with placeholders is that the information on the word’s plural vs. singular form is hidden away from the translator. So if the work that is hidden is “compilers”, the surrounding text in the translation might refer to it in singular, and that makes the sentence sound a bit off.

In some cases, the translator can deduce it from the surrounding words (e.g. if “is” or “are” is used in reference to it), but sometimes there are no hints. Luckily, the plural-singular thing isn’t very present in Chinese, Japanese and Korean, so the effect of this ambiguity is expected to be small. Try, for example to translate and back-translate “He gave me the books” with these languages, and you get “he gave me a book” — the indication for plural is lost. But there’s also a backside to this: The fact that the original word in English appears in its plural form will probably feel uneasy to an East Asian reader. I’m not sure about this, but it appears like they would use the English word in singular form anyhow, even if it refers to several pieces of whatever it is. So any use of plural will probably feel wrong to them.

Surprisingly, this can be fixed by using a placeholder like X205Xs (with the “s” in the end). This appears to be translated correctly into plural, and even the possessive form (e.g. X205Xs’) seems to work well into Hebrew.

But this hack creates a new problem: The translation might add suffixes and other grammatical elements to mark the plural form of the hidden word. If this happens, there will create a double plural. In German, for example, there are many ways to go from singular to plural, so this extra “s” just remains, when it comes after an XnX placeholder. If it isn’t removed, the result is “compilerss” (with a double “s” at the end). In Norwegian, it may add “-er” for plural (with the dash).

OK, so remove anything alphanumeric that comes after a placeholder, so that if the “s” remains, it’s gone? That may not work well either. For example, the possessive form in Swedish is expressed with a “:s” suffix and “:n” in Finnish (at least on a placeholder), so removing suffixes blindly takes its toll as well.

So even though appealing, there “s” method won’t work as a clean way to hint that the word is plural, in particular because the placeholder might get conjugated into plural in the translation. And there’s no catch-all solution for getting rid of this possible conjugation.

Given that the problem with plural is a relatively minor nuisance, that happens only when the context doesn’t say that it’s plural, it’s not worth the risk of adding garbage characters, or mistakenly removing relevant conjugation characters.

On the wishlist: The possibility to tell the translator that a blob is a noun in plural. Actually, wouldn’t it be nice to be able to do that with verbs as well, saying which tense and person?

Placeholders and Korean particles

In English, we have this thing that we say “a book” and “an orange”. The choice of the indefinite article, “a” or “an”, depends on whether the word that comes after it starts with a vowel or consonant sound.

In Korean, there are particles that are added after a noun to mark if it’s the subject, the topic or the object in the sentence. The particle is chosen according to whether the preceding word ends with a consonant or a vowel, respectively:

• Topic particles: 은 or 는 (eun or neun)
• Subject particles: 이 or 가 (i or ga)
• Object particles: 을 or 를 (eul or leul)

Not surprisingly, the particles that come after a vocal begin with a consonant, so there’s always a consonant in the game. Same principle as English’ indefinite article.

And here’s the crux: When a placeholder is used instead of a noun, Google Translate gets XnX instead of the real word, so the particle is chosen according to the “word” at hand.

So “I read the book” is translated by Google to 난 책을 읽는다 (book is 책, chaeg, ends with a consonant, hence the choice of the object particle 을, eul). But if “book” is replaced with “X10X”, we get 나는 X10X를 읽었다. “X” sounds like “eksae” in Korean, so it ends with a vowel, hence the 를 particle was used. (The word that means “I” changed from 난 to 나는, but the former is just a contraction of the latter, so it’s like “I’m” vs. “I am”)

This can be fixed automatically by looking for these particles: They are always immediately after a placeholder, and there’s a whitespace after them. The tricky part is to identify whether the replaced word ends with a consonant or a vowel, the way it’s pronounced in Korea (which may be different from the English pronunciation?).

The possessive particle, 의, as well as several other particles are indifferent to this matter.

It doesn’t seem like there’s a similar problem with Japanese nor Chinese, but I reached that conclusion based upon not finding anything related with a Google search. I will be really surprised if there was anything like this in Chinese because its script is generally unrelated to pronunciation. But with Japanese, I’m not that sure.

Maybe use a word in the target language?

I haven’t experimented a lot on this option, but maybe it will work: If a text is translated into Hebrew, and there is a Hebrew word in the middle of the text, it’s used correctly in the translation. So for example, “I ran back to בית quickly” is translated to “רצתי בחזרה לבית במהירות”. This isn’t perfect (הביתה would have been better) but it shows that a word in Hebrew is conjugated slightly and correctly.

So this opens for the possibility to replace technical terms with their relevant word in the target language. It seems like the grammar in CJK languages is exceptionally forgiving regarding nouns: There is generally no plural form, and it also seems like other conjugations are made with separate words (e.g possessive form).

Even more interesting, it works with verbs as well. “I רץ back to בית quickly” translated into “אני חוזר מהר לבית” which means “I quickly return home”. The word for “run” (רץ) was magically replaced with “return”, which is an interesting interpretation.

So maybe this can work. Not sure how much it improves, though.

So this is the small function I wrote for creating a plot and a thumbnail:

function []=toimg(fname, alt)

grid on;

saveas(gcf, sprintf('%s.png', fname), 'png');
print(gcf, sprintf('%s_thumb.png', fname), '-dpng', '-color', '-S280,210');

disp(sprintf('<a href="/media/%s.png" target="_blank"><img alt="%s" src="/media/%s_thumb.png" style="width: 280px; height: 210px;"></a>', fname, alt, fname));

The @alt argument becomes the image’s alternative text when shown on the web page.

The call to saveas() creates a 1200x900 image, and the print() call creates a 280x210 one (as specified directly). I take it that print() will create a 1200x900 without any specific argument for the size, but I left both methods, since this is how I ended up after struggling, and it’s better to have both possibilities shown.

To add some extra annoyment, toimg() always plots the current figure, which is typically the last figure plotted. Which is not necessarily the figure that has focus. As a matter of fact, even if the current figure is closed by clicking the upper-right X, it remains the current figure. Calling toimg() will make it reappear and get plotted. Which is really weird behavior.

The apparently only way around this is to use figure() to select the desired current figure before calling ioimg(), e.g.

>> figure(4);

The good news is that the figure numbers match those appearing on the windows’ titles. This also explains why the numbering doesn’t reset when closing all figure windows manually. To really clear all figures, go

>> close all hidden

Other oddities

• ginput() simply doesn’t work. The workaround is to double-click any point (with left button) and the coordinates of this point are copied into the clipboard. Paste it anywhere. Odd, but not all that bad.
• Zooming in with right-click and then left-click doesn’t affect axis(). As a result, saving the plot as an image is not affected by this zoom feature. Wonky workaround: Use the double-click trick above to obtain the coordinates of relevant corners, and use axis() to set them properly. Bonus: One gets the chance to adjust the figures for a sleek plot. If anyone knows how to save a plot as it’s shown by zooming, please comment below.