This is a minimal HOWTO on installing Linux Malware Detect for occasional use as a regular non-root user. Not that I’m so sure it’s worth bothering, given that contemporary exploit code seems to be able to go under its radar.

Background

One not-so-bright afternoon, I got a sudden mail from my web hosting provider saying that my account has been shut down immediately due to malware detected in my files (citation is slightly censored):

Hello,
Our routine malware scanner has reported files on your account as malicious. Pasted below is the report for your confirmation. Your account hosts old, outdated and insecure scripts which needs to be updated asap. Please reply back to this email so that we can work this out.

====================================
HOST: ——-
SCAN ID: 151230-0408.31792
STARTED: Dec 30 2015 04:08:40 -0500
TOTAL HITS: 1
TOTAL CLEANED: 0

FILE HIT LIST:
{HEX}php.base64.v23au.185 : /home/——/public_html/modules/toolbar/javascript21.php => /usr/local/maldetect/quarantine/javascript21.php.295615562
===============================================

I was lucky enough to have a backup of my entire hosted subdirectory, so I made a new backup, ran

$ find . -type f | while read i ; do sha1sum "$i" ; done > ../now-sha1.txt

on the good and bad, and then compared the output files. This required some manual cleanup of several new PHP files which contained all kind of weird stuff.

In hindsight, it seems like the malware PHP files were created during an SQL injection attack on Drupal 7 back in October 2014 (read again: an SQL injection attack in 2014. It’s as if a malaria breakout would occur in Europe today). The web host did patch the relevant file for me (without me knowing about it, actually), but only a couple of days after the attack broke loose. Then the files remained undetected for about a year, after which only one of these was nailed down. The malware PHP code is clearly crafted to be random, so it works around pattern detection.

Now, when we’re convinced that Linux Malware Detect actually doesn’t find malware, let’s install it.

Installing

There are plenty of guides on the web. Here’s my own take.

$ git clone https://github.com/rfxn/linux-malware-detect.git

For those curious on which revision I’m using:

$ git rev-parse HEAD
190f56e8704213fab233a5ac62820aea02a055b2

Change directory to linux-malware-detect/, and as root:

# ./install.sh
Linux Malware Detect v1.5
            (C) 2002-2015, R-fx Networks <proj@r-fx.org>
            (C) 2015, Ryan MacDonald <ryan@r-fx.org>
This program may be freely redistributed under the terms of the GNU GPL

installation completed to /usr/local/maldetect
config file: /usr/local/maldetect/conf.maldet
exec file: /usr/local/maldetect/maldet
exec link: /usr/local/sbin/maldet
exec link: /usr/local/sbin/lmd
cron.daily: /etc/cron.daily/maldet
maldet(15488): {sigup} performing signature update check...
maldet(15488): {sigup} could not determine signature version
maldet(15488): {sigup} signature files missing or corrupted, forcing update...
maldet(15488): {sigup} new signature set (2015121610247) available
maldet(15488): {sigup} downloading http://cdn.rfxn.com/downloads/maldet-sigpack.tgz
maldet(15488): {sigup} downloading http://cdn.rfxn.com/downloads/maldet-cleanv2.tgz
maldet(15488): {sigup} verified md5sum of maldet-sigpack.tgz
maldet(15488): {sigup} unpacked and installed maldet-sigpack.tgz
maldet(15488): {sigup} verified md5sum of maldet-clean.tgz
maldet(15488): {sigup} unpacked and installed maldet-clean.tgz
maldet(15488): {sigup} signature set update completed
maldet(15488): {sigup} 10822 signatures (8908 MD5 / 1914 HEX / 0 USER)

Reduce installation

Remove cronjobs: First /etc/cron.d/maldet_pub

*/10 * * * * root /usr/local/maldetect/maldet --mkpubpaths >> /dev/null 2>&1

and also /etc/cron.daily/maldet (scan through everything daily, I suppose):

#!/usr/bin/env bash
export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH
export LMDCRON=1
. /usr/local/maldetect/conf.maldet
if [ -f "/usr/local/maldetect/conf.maldet.cron" ]; then
	. /usr/local/maldetect/conf.maldet.cron
fi
find=`which find 2> /dev/null`
if [ "$find" ]; then
	# prune any quarantine/session/tmp data older than 7 days
	tmpdirs="/usr/local/maldetect/tmp /usr/local/maldetect/sess /usr/local/maldetect/quarantine /usr/local/maldetect/pub"
	for dir in $tmpdirs; do
	 if [ -d "$dir" ]; then
	  $find $dir -type f -mtime +7 -print0 | xargs -0 rm -f >> /dev/null 2>&1
	 fi
	done
fi

if [ "$autoupdate_version" == "1" ] || [ "$autoupdate_signatures" == "1" ]; then
	# sleep for random 1-999s interval to better distribute upstream load
	sleep $(echo $RANDOM | cut -c1-3) >> /dev/null 2>&1
fi

if [ "$autoupdate_version" == "1" ]; then
	# check for new release version
	/usr/local/maldetect/maldet -d >> /dev/null 2>&1
fi

if [ "$autoupdate_signatures" == "1" ]; then
	# check for new definition set
	/usr/local/maldetect/maldet -u >> /dev/null 2>&1
fi

# if we're running inotify monitoring, send daily hit summary
if [ "$(ps -A --user root -o "cmd" | grep maldetect | grep inotifywait)" ]; then
        /usr/local/maldetect/maldet --monitor-report >> /dev/null 2>&1
else
	if [ -d "/home/virtual" ] && [ -d "/usr/lib/opcenter" ]; then
		# ensim
	        /usr/local/maldetect/maldet -b -r /home/virtual/?/fst/var/www/html/,/home/virtual/?/fst/home/?/public_html/ 1 >> /dev/null 2>&1
	elif [ -d "/etc/psa" ] && [ -d "/var/lib/psa" ]; then
		# psa
		/usr/local/maldetect/maldet -b -r /var/www/vhosts/?/ 1 >> /dev/null 2>&1
        elif [ -d "/usr/local/directadmin" ]; then
                # DirectAdmin
                /usr/local/maldetect/maldet -b -r /home?/?/domains/?/public_html/,/var/www/html/?/ 1 >> /dev/null 2>&1
	elif [ -d "/var/www/clients" ]; then
		# ISPConfig
                /usr/local/maldetect/maldet -b -r /var/www/clients/?/web?/web 1 >> /dev/null 2>&1
	elif [ -d "/etc/webmin/virtual-server" ]; then
		# Virtualmin
                /usr/local/maldetect/maldet -b -r /home/?/public_html/,/home/?/domains/?/public_html/ 1 >> /dev/null 2>&1
	elif [ -d "/usr/local/ispmgr" ]; then
		# ISPmanager
		/usr/local/maldetect/maldet -b -r /var/www/?/data/,/home/?/data/ 1 >> /dev/null 2>&1
	elif [ -d "/var/customers/webs" ]; then
		# froxlor
		/usr/local/maldetect/maldet -b -r /var/customers/webs/ 1 >> /dev/null 2>&1
	else
		# cpanel, interworx and other standard home/user/public_html setups
	        /usr/local/maldetect/maldet -b -r /home?/?/public_html/,/var/www/html/,/usr/local/apache/htdocs/ 1 >> /dev/null 2>&1
	fi
fi

And then remove the bootup hooks (I could and should have done this with chkconfig, actually):

# rm `find /etc/rc.d/ -iname S\*maldet\*`
rm: remove symbolic link `/etc/rc.d/rc3.d/S70maldet'? y
rm: remove symbolic link `/etc/rc.d/rc4.d/S70maldet'? y
rm: remove symbolic link `/etc/rc.d/rc2.d/S70maldet'? y
rm: remove symbolic link `/etc/rc.d/rc5.d/S70maldet'? y

Configuration

Edit /usr/local/maldetect/conf.maldet. The file is self-explained. The defaults are quite non-intrusive (no quarantine nor cleaning by default, no user suspension etc.). I turned off the automatic updates (I don’t run this as a cron job anyhow) and opted in scans by users:

scan_user_access="1"

Other than that, I kept it as is.

Preparing for run as non-root user

As a regular user (“eli”) I went

$ maldet
touch: cannot touch `/usr/local/maldetect/pub/eli/event_log': No such file or directory
/usr/local/maldetect/internals/functions: line 31: cd: /usr/local/maldetect/pub/eli/tmp: No such file or directory
mkdir: cannot create directory `/usr/local/maldetect/pub/eli': Permission denied
chmod: cannot access `/usr/local/maldetect/pub/eli/tmp': No such file or directory
mkdir: cannot create directory `/usr/local/maldetect/pub/eli': Permission denied
chmod: cannot access `/usr/local/maldetect/pub/eli/sess': No such file or directory
mkdir: cannot create directory `/usr/local/maldetect/pub/eli': Permission denied
chmod: cannot access `/usr/local/maldetect/pub/eli/quar': No such file or directory
sed: couldn't open temporary file /usr/local/maldetect/sedIuE2ll: Permission denied

[...]

So it expects a directory accessible by non-root self. Let’s make one (as root)

# cd /usr/local/maldetect/pub/
# mkdir eli
# chown eli:eli eli

Giving it a try

Try

$ maldet -h

And performing a scan (checking a specific sub-directory on my Desktop):

$ maldet -a /home/eli/Desktop/hacked/
sed: couldn't open temporary file /usr/local/maldetect/sedcSyxa1: Permission denied
Linux Malware Detect v1.5
            (C) 2002-2015, R-fx Networks <proj@rfxn.com>
            (C) 2015, Ryan MacDonald <ryan@rfxn.com>
This program may be freely redistributed under the terms of the GNU GPL v2

ln: creating symbolic link `/usr/local/maldetect/sigs/lmd.user.ndb': Permission denied
ln: creating symbolic link `/usr/local/maldetect/sigs/lmd.user.hdb': Permission denied
/usr/local/maldetect/internals/functions: line 1647: /usr/local/maldetect/tmp/.runtime.hexsigs.18117: Permission denied
maldet(18117): {scan} signatures loaded: 10822 (8908 MD5 / 1914 HEX / 0 USER)
maldet(18117): {scan} building file list for /home/eli/Desktop/hacked/, this might take awhile...
maldet(18117): {scan} setting nice scheduler priorities for all operations: cpunice 19 , ionice 6
maldet(18117): {scan} file list completed in 0s, found 8843 files...
maldet(18117): {scan} scan of /home/eli/Desktop/hacked/ (8843 files) in progress...
maldet(18117): {scan} 8843/8843 files scanned: 0 hits 0 cleaned
maldet(18117): {scan} scan completed on /home/eli/Desktop/hacked/: files 8843, malware hits 0, cleaned hits 0, time 253s
maldet(18117): {scan} scan report saved, to view run: maldet --report 151231-0915.18117

Uh, that was really bad. The directory contains several malware PHP files. Maybe the signature isn’t updated? The file my hosting provider detected was quarantined, and those that were left are probably sophisticated enough to go under the radar.

Update the signature file

Since I turned off the automatic update of signature files, I have to do this manually. As root,

# maldet -u
Linux Malware Detect v1.5
            (C) 2002-2015, R-fx Networks <proj@rfxn.com>
            (C) 2015, Ryan MacDonald <ryan@rfxn.com>
This program may be freely redistributed under the terms of the GNU GPL v2

maldet(15175): {sigup} performing signature update check...
maldet(15175): {sigup} local signature set is version 2015121610247
maldet(15175): {sigup} latest signature set already installed

Well, no wonder, I just installed maldet.

So the bottom line, mentioned above, is that this tool isn’t all that effective against the specific malware I got.

In other words, if I wrote %mail{1}% somewhere in the page’s source, that is replaced by the second email address in a known list of addresses.

Of course there the Token module. But I wanted the replacement to depend on the token string. Globally.

I use arthemia, so this appears everywhere in the paths to files.

Bad attempt #1

I thought that dvessel’s suggestion on a pretty old page would do the job (and it looks like it was obvious to everyone which file to edit, sites/all/themes/arthemia/template.php). But that suggestion was good for Drupal 5.

The theme I used, arthemia, already has an arthemia_preprocess_page() function, but not an arthemia_render_template(). I followed the suggestion, and nothing happened, of course. Drupal 7 doesn’t look in that direction.

Bad attempt #2

There is this golden rule in Drupal, that you don’t edit the core functions. Well, well, it was time to break it again. This time with the core of the core: include/common.inc, modifying drupal_render. Yup, I couldn’t have broken that rule harder. But after a few hours of the regular Drupal ceremony of reading a lot of useless tips on the web, I thought it was best to just get the job done.

The main problem was that all hooks are related to a certain type of elements. There is no hook for “just before the HTML is handed over to the browser”. Or that’s the impression I got.

First, I added the script I wanted included in every page. It would prefer it to be in the <head> section (because that’s usually where global scripts are) and I would prefer it to be a piece of content. But again, hours of looking for the “Drupal” way to do it, I just added the following to the top of sites/all/themes/arthemia/page.tpl.php:

<script language="JavaScript">
<!--
function myfunction(i) {
 [ ... ]
}
-->
</script>

It’s an ugly solution, but it gets the job done. I could, in theory, plant it as the content of an unused part of the page, and then mangle the page’s structure as it was being processed. Too much work for the cute shortcut of altering the script from Drupal’s interface.

And now to finding the markups, and make them run the function above. Using the preg_replace_callback() function, render_template was edited to this in the end of the function (just before “return $output”):

// Mangle
 $output = preg_replace_callback("|%mail\{(\d+)\}%|",
   function ($matches) {
     return "<script language=\"JavaScript\">\n<!--\nmyfunction($matches[1]);\n-->\n</script>";
   },
   $output)

That worked well. Actually, too well. The replacement took place also in editing boxes, so the original markup text couldn’t be edited.

The correct way

OK, so I took one step back. Let’s use the regular hooking mechanism, after all, and define the hooks for the content and the footer. Good enough. And as a side effect, the core is left intact, as I only edit the theme’s page template file.

So instead of the changes above I added this to the top of sites/all/themes/arthemia/page.tpl.php:

<script language="JavaScript">
<!--
function myfunction(i) {

 [ ... ]
}
-->
</script>

<?php
function mail_post_render($content, $element) {
  $content = preg_replace_callback("|%mail\{(\d+)\}%|",
    function ($matches) {
      return "<script language=\"JavaScript\">\n<!--\nmyfunction($matches[1]);\n-->\n</script>";
    },
    $content);

 return $content;
}

if ($page['content']) {
 $page['content']['#post_render'] = array('mail_post_render');
}
if ($page['footer']) {
 $page['footer']['#post_render'] = array('mail_post_render');
}

?>

And that finally worked in a sane manner: The footer and content were mangled, and I was still able to edit the original text. Hurray!

As usual, a simple task took a full day to complete. Welcome to the wonderful world of Drupal.

I needed some neat overlay to appear when the user clicks those question-mark icons on a specific, Drupal-based page. So I went for using jQuery’s tooltip widget, with due adaptations. It turned out to be a wise choice.

The idea is simple: Click on the question mark, an overlay window appears right underneath it, and stays there. Click on the “X” at the window’s top right, the window closes. Simple and clean.

These are my notes as I set this up. Most of the work is around the fact that tooltips usually appear and disappear on mouse hovering, not clicks.

Getting a bundle for a tooltip

Build your own set and download on jQuery’s official site. For a plain tooltip activation, pick the following options: All UI Core, no Interactions. Widgets: Only Tooltip. Effects: Only Effects Core and Fade Effect.

From the downloaded bundle, the following inclusions (in the <head> section) suffice (version numbers may vary):

<style>@import url("jquery-ui-1.10.0.custom.css");</style>

and

<script type="text/javascript" src="jquery-1.9.0.js"></script>
<script type="text/javascript" src="jquery-ui-1.10.0.custom.min.js"></script>

The “.min” version is just a condensed version of the non-min JavaScript file. The rest of the bundle can be ignored for a minimal tooltip implementation.

In a Drupal environment

Since Drupal uses jQuery as well, the initial idea was to rely on the script loaded anyhow, but I got the “$ is not a function” error in the error console. To solve this, the script inclusions shown above should appear after Drupal’s. In theory, the jQuery.noConflict() call should be used to save the previous “$” function, but since Drupal doesn’t inject jQuery calls after its last declaration in <head>, there is no need for this. This is OK for me, since I work on one specific page, but it’s theoretically possible that other Drupal pages put JavaScript in the <body>. So keep this in mind.

Maybe the best way is to load your own jQuery script bundle before Drupal’s, run noConflict() and then load Drupal’s. Just to be safe.

IE6 note

Internet Explorer 6 is an ancient thing, but since I always check against antique browsers, I noted that if there’s a selection box under the tooltip, it will appear above the tooltip, despite the latter’s z-index set to 9999. This is a bug specific to IE6, and therefore nothing to worry about too much. Anyone with IE6 is used to pages appearing ugly anyhow.

Declaring the jQuery handlers and hooks

This is a general note on how to inject jQuery definitions into the HTML.

Following jQuery’s recommendation, the application-specific definitions are made in the <head> section, in an anonymous function which is triggered by the “ready” event. This makes the jQuery hooks up and running as soon as the DOM hierarchy is set up, but doesn’t requires the onload event, which is triggered only when all assets (e.g. images) have been loaded.

So to run a tooltip on all elements marked “myclass” (they need to have a title= assignment),

<script>
$(function() {
 $( ".myclass" ).tooltip();
});
</script>

The hooking to the “ready” event is implicit by the $(function() { … }) declaration.

Appear on click, don’t disappear

To make a “Hello world” tooltip appear on clicking an image of class “helpimg”, this goes into the <head> section (partly taken from here):

<script>
$(function() {
 $( ".helpimg" ).tooltip({
   content: "<i>Hello, world</i>",
   items: 'img'
 })
 .off( "mouseover" )
 .on( "click", function(){
    var clicked = this;
    $( ".helpimg" ).each(function (i) {
      if (this != clicked) {
         $( this ).tooltip( "close" );
      }
    });

    $( this ).tooltip( "open" );
    $( this ).unbind( "mouseleave" );
    return false;
 });

 $( "body" ).on( "click", ".helpclose", function(){
   $( ".helpimg" ).each(function (i) { $( this ).tooltip( "close" ); });
   return false;
 });
});
</script>

This little script does two things: Assign a tooltip object for each DOM element of class “helpimg” (for opening the tooltip) and an event handler to each “helpclose” element (for closing it).

See the tooltip API reference for the meanings of the attributes (content, item etc.).

Open the tooltip (.helpimg assignment)

The “content” attribute obviously sets the tooltip’s content. Here it’s given as a string itself, which is OK if the data is known when the tooltip is generated. Another possibility is to put a function here. Unlike what some documentation implies, there is no bodyHandler attribute (on jquery-ui-1.10.0). To provide a callback, a function is given instead of the string. For example, to put the ID of the element clicked on in the tooltip box, it should say (replacing the content: assignment above):

content: function() {
  return $( this ).attr( "id" );
},

The “item” attribute defines the item type(s) involved. If this is absent, the click operation works only once: After the tooltip fades away, another click on the same item do nothing. Frankly, I don’t understand why this is.

The anonymous function defined in the .on() call is called when the element is clicked. Its first part (the definition of “clicked” and the “each” call) scans all elements of class “helpimg” and sends them a “close” command. This part is optional, to maintain an exclusive tooltip open. The if-statement in the each-function prevents closing the tooltip that is just about to be opened, to prevent flickering. Maybe it’s better to let the flickering happen, so that the user gets some feedback on the click. A matter of taste, I suppose.

Note that the function defined in the “each” call has a variable, which is the index. It’s not used in this case.

The call to unbind() (marked in red) kills the “mouseleave” event handler(s) that are related to the tooltip, so it doesn’t listen to this event. Normally, the “open” method sets an event handler for either “mouseleave” or “focusout”, depending on whether the triggering event was “mouseover” or “focusin”, respectively.

There is a non-specific call to unbind(), which makes the tooltip deaf to any event, but that made the tooltip appear only on the first click and then not again. So obviously there are some events handlers that should be left (maybe timers?).

Closing the tooltip

Since the tooltip won’t close by itself, this must be done by clicking on some element. The dedicated elements (actually, top-right “X” boxes) are given the “helpclose” class. When one of those elements get the “onclick” event, all tooltips are closed, with a function identical to the one used before opening a tooltip.

The important difference is the selection of these elements. Since they are hidden when the script is run, selecting them with $( “.xillyclose”).on( “click”, …) will catch nothing. The solution is to delegate the event to the “body” object, which is of course visible, and push the selector to within the .on() call. This way, the to-be visible elements are selected as they appear.

Closing on ESC keystroke

I found myself hitting ESC to close a tooltip, so why not support it? The idea is to catch keyup events, check if the key code happens to be 27 (ESC), and act as necessary.

$( document ).on( "keyup", function(event){
 if (event.keyCode == 27) {
 $( ".helpimg" ).each(function (i) { $( this ).tooltip( "close" ); });
 return false;     
}

I found the key code for ESC by running the demo on the .keyup() API page. Note that the .keyup() command is just a shorthand for the .on() call used above.

Position the tooltip

To get the tooltip properly positioned, the position property is set up during initialization. The declaration above should therefore start as

$(function() {
  $( ".helpimg" ).tooltip({
    content: "<i>Hello, world</i>",
     items: 'img'
     position: {
       my: "center top",
       at: "center bottom+5",
       collision: "fit",
       within: "#content"
     }
[ ... ]

To avoid confusion, let’s think about the tooltip as a piece of paper that we want to nail on a billboard.

As described in the docs, the “my” property says where the on the tooltip we should place that nail (on the horizontal center and top of the tooltip), and the “at” property says where we should stick that nail (the horizontal center and bottom of the object is relates to, just 5 pixels lower).

The “collision property” tells what to do when the positioning would make the tooltip exceed the given area (“fit” means push it away from the border as necessary, but don’t throw it to the opposite side). “within” defines the framework in which the tooltip should be placed (the id=”content”, which is the main context box, defined by Drupal).

Conclusion

It took some time to get the hang of it, but all in all, jQuery is worth it, in particular by making the application code compact and therefore less bug prone.

It also handles those corner cases gracefully and takes care of browser compatibility. After all, what’s good for Wikipedia is good for me.

Using Drupal views basically is trying to figure out how to trick the machine into making the SQL query I would have written in five minutes. As a matter of fact, I don’t think I had a chance of getting this right, hadn’t I known SQL pretty well.

Or, as one of the help pages was kind enough to clarify (the left side is merely examples):

SELECT n.title, u.name <–> fields
FROM {node} n base table <–> view type
INNER JOIN {users} u ON n.uid = u.uid <–> relationship
WHERE n.status = 1 <–> filter
AND u.uid = arg(1) <–> argument
ORDER BY n.changed DESC <–> sort

and I’ll add:

WHERE n.nid = p.nid <–> Contextual filter
GROUP BY <–> Aggregation

A word about relationships: The relationship entries in the Views page define the “ON” part in the JOIN. To access the values of the fields in the joined tables (e.g. as a displayed field), just select the desired value in the first menu (e.g. “add field”), and then pick the administrative name in the “relationship” drop-down menu. For some reason, I had the initial expectation that the “ON” value would appear in itself in the field list, but it doesn’t, since it’s available just as any value, only picked from the respective table.

Recommended modules

The Development module (“devel”) allows a dump of all SQL queries made while producing the currently shown page. It’s recommended just to get an idea of what a database hog Drupal is. The second, very recommended module is PHP views, despite its development release status. The latter allows injecting small pieces of PHP exactly where they are needed. In particular, there’s the PHP:Global pseudofield for both display and sorting, so rather than banging the head on how to twist things around, just write a small and elegant snippet in PHP.

Injecting a view with PHP

Possibly using the PHP text format, just type in any node’s text input (useful for making a view within a book node):

<?php print views_embed_view('doc_outline', $display_id = 'default') ?>

where ‘doc_outline’ is the machine name for the view (as it appears in URLs related to the view) and ‘default’ could be exchanged with ‘page’ or ‘block’, but why bother if the view has a single format? See the API page.

There has also been suggestions about embedding a block in PHP, but I never tried that.

Strategy

The purpose: Making a “relevant pages” block, based upon common tags of the pages. Don’t tell me Drupal has something doing that, because I know. It’s just that I’ve learned that these easy solutions always end up with more work. Besides, I have a special thing I want.

The setting is as follows: I have a special taxonomy named “relevance terms”. Pages with common terms in this taxonomy are considered to have similar content. I didn’t use the original tags, because I may want to use them for something representative.

Also, each content page has an extra field “LinkText”, which contains the text to appear on links to the page. For example, the download page’s title is simply “Download” but the link to this page should say something more imperative.

The immediate (and wrong) way to go is to make a view of content titles. Without any filtering, you get all content pages. So use the context filter to get only the current page, and use relations to list all taxonomy terms. Now another relation filter to expand this to pages for each taxonomy term? But the context filter kills any other page than the currently displayed. It’s a dead end.

So the right way is to make a view of the page’s taxonomy terms. For each term, list the pages using it, and then squash duplicates. And then make the output nice. Easier said than done.

First steps

Add a new View, showing Taxonomy Terms of type Relevance Taxonomy. Don’t create a page, but a block. Display as an HTML list of fields. Save & Exit, and insert the block somewhere in the page, so it can be tested. Previews won’t work here, because it runs  on Taxonomy nodes, not pages. Set title and such.

The number of items should be limited, and I don’t use a pager.

Contextual Filter & Relationship

This is done first, so we don’t mess with aggregation, which is going to be set up pretty soon. Under Advanced, add a contextual filter on “Taxonomy Term ID”. The following window will complain that there’s no source for contextual filter, so a default must be supplied. This is because we’re running in block context. The source is taken from the page.

We want the node ID to be compared with the current page, so pick “provide default value” and “Taxonomy term ID from URL”. Uncheck “Load default filter from term page” but check “Load default page from node page” and also “Limit terms by vocabulary” and pick the Relevance Taxonomy as the chosen vocabulary. Under “More” check “Allow multiple values” This is necessary, since we don’t want just the first term to be used. I’m not sure if this item appears without setting up relationships. So if it’s missing, set up a relationship and come back to add this.

That’s it. Save and check up. We should now have a simple list of relevance terms in the view.

Next we add a relationship with the pages having the terms: Check “Taxonomy term: Content using relevance” (note that “relevance” is the name of the vocabulary here), check “Require this relationship” on the next screen (I suppose this makes an INNER JOIN as opposed to a LEFT JOIN), and save this.

Checking where we stand, we have each taxonomy term appearing a number of times. This is the natural behaviour of an inner join: Each combinations of terms and pages using them creates a line. Since the pages aren’t listed, we just see each term repeated.

And since we’re at it, let’s eliminate the shown page’s entry in the related pages’ list. We need negative contextual filtering here: So add a new contextual filter, check “Content: Nid” (it wasn’t there until we added the relationship). Provide default value as “Content ID of URL”, and under “More” check “Exclude”. So if the current page matches the shown page, it’s not shown.

Save and verify than one or a few items have disappeared from the list.

Aggregation

Aggregation is “GROUP BY” in SQL, meaning that several rows with the same value in one of the fields are turned into a single row. Instead of the field’s value we have the count of rows grouped together or the maximum, minimum, average value or whatever the database allows. Aggregation is needed to eliminate the duplicate rows created by the relationship (that is, the inner join).

True, there is a “Distinct” checkbox under “Query settings” but it’s ineffective, since each of these duplicate rows are indeed distinct when the database answers the query. What makes them duplicate is the fact that the taxonomy term is dropped in the display. “Distinct” just adds the DISTINCT word to the SQL query.

So at this point, change “Use aggregation” to Yes. Things will get slightly messier from this point on.

Adding fields

Rule #1 for fields: The order they appear matters. In particular when using rewrite rules: Inserting data from other fields in substitution patterns works only for fields above the displayed one (those declared before).

Remember that the goal is to show the Linktext field as a link to the page, and not just the title.

So the first field to add is the Node’s path (aliased link). We will use it later on. In the list, check “Content: Path”. Under “Aggregation type” pick “Group results together” which is what we pick all the time if not for any special reason. This choice wouldn’t appear without enabling aggregation, of course. On the third and last window, check “Exclude from display” unless you want to see it for debugging.

The second field to add is the link text. In the list, check “Content: LinkText”. Under “Aggregation type” pick “Group results together” and pick the “Entity ID” as group column, and no additional ones.

On the third page uncheck “Create a label” (no “LinkText:” shown to user). Under “Rewrite results” check “Output this field as a link”. Write [path] in the Link path text box. This string can be found in the Replacement Pattern list just below. The path was there because it was defined before the current field.

Check “Use absolute path” or the links start with a double-slash and render useless.

At this point I’ll mention that it’s possible to insert arbitrary HTML with replacement patterns. So it’s really useful.

Squashing duplicates

At this point it’s pretty evident that we have double entries, and these taxonomy terms should be removed.

So it’s time to edit the first field: The “Taxonomy Term: Name” and check “Exclude from Display”. But even more important, enter “Aggregation settings” and change “Aggregation type” to “Count”. The magic is that instead of a row for each taxonomy term, we get a single row with the number of them, ending up with a single row for each link.

Filter out inaccessible items

As is, users will see links to items the user can’t access. So let’s add a simple filter (“Filter Criteria”). Pick “Content: Published or admin” and “Group results together” in the two following menus. And then just apply on the next menu. Done.

Note that unpublished items will still appear for admins, since the criterion is access. Pick “Content: Published” and choose “Yes” to remove unpublished items for anyone.

Sorting

I have to admit that I failed on this one at first. My intention was to sort the output depending on the number of mutual tags. That would be easy in SQL, since the COUNT() can be applied a label with the AS keyword. I found nothing to assist this in the Views menus.

As it turned out, values of COUNT() are available, but not through the menu interface. With the Views PHP module, it’s a piece of cake.

Say that there’s already a field saying “COUNT(Taxonomy term: Name)”, then add a sort criteria of type global PHP and set the code to

return ($row2->name - $row1->name);

Since $row1->name is the count of the rows with the same name field, this simple chunk of code does the work.

Using the view in a non-Drupal page

Sometimes the whole framework is just too heavy for a site, and all that’s needed is just a view in a plain PHP file. So if a view block with the name “inject” exists in the system, the following code snippet displays it (and a lot of CSS mumbo-jumbo).

<?php

chdir('trydrupal');

define('DRUPAL_ROOT', getcwd());

require_once DRUPAL_ROOT . '/includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

print views_embed_view('inject', $display_id = 'default')
?>

Note the chdir(). It allows Drupal to be installed in a completely separate directory (“trydrupal” in this case).

March 2019 update: Drupal was a huge mistake. Just in case someone out there still has a chance to escape.

And now I appreciate how easy it is to install and use WordPress. Drupal is definitely not good for my health: Hours of frustration trying to accomplish seemingly simple things can’t do me good. It’s tempting to conclude, that Drupal contributors have some job-security-by-obscurity kind of mind set. And I do understand why they end up with nasty hacks in PHP: The nice menus offer a million features, except what is really needed.

General remarks

• The development module is a blessing. In particular the ability to display the SQL queries related to a page: Under Modules, go to “configure” for the Development Module and check “Display query log”. And there’s an “A” next to each row of SQL, which means “display argument”. That is, show the query with what was inside the placeholders. Soooo nice.
• The “devel” tab (part of the development module?) is great: The metadata, with internal names is all listed there. Everything there is to know about a node.
• Enable the PHP text format. It allows running PHP code within any node, which is sometimes necessary.
• The number of files necessary to display a simple page is absolutely insane: A simple textual home page reaches 46 files (many of which are Javascript and CSS files) with a total of 205 kB (!). No wonder the site is slow.
• Take a look on the module list which is sorted by number of installations by default. The things you want are most likely in the beginning of this list.
• Use URL aliases for every post. A node number says nothing and it’s a clear loss of Google juice.
• CKEditor allows the insertion of a DIV container, for which, among others, the language direction can be chosen. This is a great thing for Hebrew, Arabic and other right-to-left languages.
• Drupal’s taxonomy is hierarchical, so the site’s hierarchy can be implemented by setting up a hierarchy of terms within a certain vocabulary, and then tag nodes (pages) in order to place them in the menu hierarchy.
• Flush the entire cache every now and then when changing things.
• The administrative menu gets less quirky as time goes by. Go figure.
• Structure > Blocks is where blocks are added to or removed from the site globally.
• Special pages are best done as Panels. Note that a panel page is just like a normal page with all the blocks around it.
• Special note to self: I’ve hacked modules/system/html.tpl.php (see below). A Drupal upgrade will require rehacking.

Milestones in setting up the site:

• Enable clean URLs
• Enable book module
• Installing a rich text editor (duh…). I went for the classic WYSIWYG module (see below)
• Remove the “powered by Drupal” footer (with all due respect)
• Themes go to tar -xzf in sites/all/themes. Switch themes through Appearance (not any submenu)
• Downloaded and installed (tar -xzf in sites/all/modules) pathauto,  Views,  Taxonomy Menu, (CCK is part of Drupal 7),  Administration Menu, IMCE (image uploader, I’ll do that manually), Backup and Migrate, and Panels. Ctools was installed as well for Panels. Same goes with Token for pathauto. And then devel. And Disable messages. (Page Title was downloaded but not used to, since it’s great for everything except what I needed: Set the title from a within a view. It’s maybe possible, but I couldn’t figure out how). Views PHP (dev version!) for setting up page title (see below).
• And then enabled the modules, of course.
• Set up cron (run once an hour). I got a lot of “access denied” in the beginning, and then I realized that I should get the link with the cron key under Reports > Status Reports. And then I found out that cron.php denies access when in maintenance mode. So start the cron thing only when the site goes online.
  Another thing about cron is that my hosting provider seems to block Curl from running cron jobs with a silly check on the user agent. My updated cron job command hence goes: curl –silent –compressed -A ‘My cronjob’ ‘http://example.com/cron.php?cron_key=(the key supplied on the status page)’
• Set the default page: Configuration > System > Site Information
• Enable statistics: Modules > List > Statistics > Configure. Set to never delete log info (I don’t expect massive traffic).
• Disable Taxonomy term pages for users (see below). Same with node-by-number.
• Ask web host to increase RAM limit (they gave me 128MB, which is OK)
• Enable the “Disable messages” module and configure it to display error messages to admins only (why isn’t this in the core?). This is done by setting permissions (the existence of the module creates new relevant entries). There is no need to configure the module further.
• Edit public_html/sites/all/modules/ctools/includes/cleanstring.inc and replace “\x{d800}” with “\x{e000}” in the expression for CTOOLS_PREG_CLASS_ALNUM. 0xd800 is apparently not a legal Unicode point, and without this change there’s a warning saying exactly that: “Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in ctools_cleanstring() (line 157 of /home/theuser/public_html/sites/all/modules/ctools/includes/cleanstring.inc).”
• Edit sites/all/themes/arthemia/template.php and change arthemia_primary() at the beginning to read

  function arthemia_primary() {
   $output = '<div id="page-bar">';
   $the_menu_tree = menu_tree(variable_get('menu_main_links_source','main-menu'));
   $output .= drupal_render($the_menu_tree);
   $output .= '</div>';
   return $output;
  }

  Originally, drupal_render was called with the return value of menu_tree(), an a strict warning is issued about calling a function expecting a reference with a non-reference. Or something.

• April 2018 update: Patch Drupal against this arbitrary code execution exploit (which, BTW, was relevant for current Drupal versions as well. Just being up to date wouldn’t have helped against this one).

Installing the WYSIWYG module

Spoiler: The clear winners are CKEditor and NicEdit. CKEditor because it has the nicest features, but for some reason it has its own spell checked which contacts a dictionary over the web rather than using the browser’s spell checking. And hence it’s disabled by default. It doesn’t make sense to tell the world what you’re typing as a default setting.

• Downloaded from here.
• tar -xzf at sites/all/modules/.
• Created a “libraries” directory under sites/all (why wasn’t it already there?)
• Downloaded CKEditor and did tar -xzf at the “libraries” just created. Be sure that CKEditor wasn’t released after Drupal, because it may not install, and the message “The version of CKEditor could not be detected” will appear at “Wysiwyg profiles”
• Downloaded NicEdit (default configuration), created a “nicedit” directory under “libraries” and unzipped the file there.
• Downloaded YUI, and unzipped it directly on “libraries”
• On web interface menu: Configuration > Text Formats > Add Text format. Add a new text format, e.g. NicEdit and set it for use only for admins (for example).
• Then on web interface menu: Configuration > Wysiwyg profiles assign NicEdit text format to NicEdit editor
• Then on web interface menu: Configuration > Wysiwyg profiles > edit NicEdit > Buttons and plugins and check all checkboxes (TAB and space bar came handy). Or the rich edit box looks like just a plain text box. Why this extra step was necessary is pretty much beyond me.
• Same procedure for other editors.

Setting up a taxonomy menu structure

The sequence below is based upon the Administration Menu module (things may be found in other places using the core interface):

• Create a new menu:  Structure > Menus > Add Menu
• Create a new vocabulary: Structure > Taxonomy > Add Vocabulary.
• Generate a few terms in the new vocabulary
• Edit the new vocabulary. Under “Taxonomy menu” pick the the new menu, and check the “Select to rebuild the menu on submit” checkbox. This copies the taxonomy items into the menu (not very sophisticated, is it).
• Now on Structure > Blocks > { your chosen theme } put the menu somewhere. Or more precisely, find the new menu in the list, and assign it a point of appearance. Don’t forget to submit the form.
• To use the menu items as top-page tabs, go to Structure > Menus > Settings and set the source as the new menu

Note that new vocabulary items don’t appear automatically on the menu, and it’s not a matter of flushing the cache or running cron. The menus need to be updated by rebuilding them as mentioned above.

So the bottom line is that I tend to trash the Taxonomy menu and use the book structure.

Tagging pages

• Go to Structure > Content Type > { The content type to alter } > Manage Fields
• Add a new field of type “Term Reference”  and select any of the widgets offered.
• Click Save settings
• On the next page, you’ll be asked to choose the vocabulary to use. Save this as well.
• On the next page set up the number of entries allowed (unlimited?) and default tag.
• You may want to hide this field in the display (if it’s for internal uses): Structure > Content Types > { The content type to alter } > Manage Display and set both label and format to hidden. Be sure to apply this to all view modes, full content in particular.

Custom styling for panes

When the GUI tools don’t do the job, editing the theme’s main CSS file does (or a local.css file in the theme directory, but yet another CSS file?). Then, in the content editing page where the panes appear, pick the inner gear menu > CSS Properties and enter a new class name, say mypane-class. In the CSS file setting the attributes for a header within the pane will look like

.mypane-class h2 {
 font: Arial, Helvetica, sans-serif;
 font-weight: bold;
 font-size: 14px;
}

Allowing injected Javascript

Full HTML actually allows Javascript, only it wraps it with CDATA. Which could have been forgiveable, had it done that correctly, and not killed the script on some browsers. The really annoying thing about this is that the purpose of this CDATA statement is merely to silence XHTML validators.

The trick is simple, and explained in a video (I prefer things written down, but anyhow). The idea is to create a new input format. The way Drupal works, it filters the content of the database during page display (the input goes in as is), so the idea is to create a new filter which does nothing. So what is in the database is shown. Not the safest thing in the world for general practice, but sometimes there’s no choice.

Basically, go to the modules list, find “Filter” and click its “configure”. Pick “add text format”, name it “Unfiltered” (or something), make sure none of the filters is checked and that only the administrator has access to this format. That’s it. Now whatever goes in, goes out. Use this format, enter HTML with Javascript and whatever.

Removing the navigation links at the bottom of a book page

As suggested here, don’t edit the original PHP code generating the links, but rather copy /public_html/modules/book/book-navigation.tpl.php to the theme’s directory, edit the copy, and flush the caches.

To retain the links in the lower hierarchy, but remove the navigation links, simply change

<?php if ($has_links): ?>

to

<?php if (0): /* ($has_links): */ ?>

(deleting code is for barbarians)

Disallow access to taxonomy pages

There may be an elegant way to do this with Drupal’s own interface, but I’m yet to discover it. So the trick is to create a custom page, and make it empty. Silly, but it works.

Under Views, clone the view which emulates the core’s taxonomy display (it’s name is simply “Taxonomy”). Add a header, saying whatever you want, which is active even when no items match (who cares). This is merely so you know it’s not the default page you’re watching.

Then set an access rule, saying only admins are allowed: Access > Role > Check administrator.

To get the same effect with nodes, do something similar for node/%. Don’t clone the taxonomy view, since they display taxonomy, and sometimes a page needs to be displayed by node (for admin purposes).

Setting the page title hack

Goal: Set up the page title (the one within <title> tags) based upon the title of a content entry within a view.

I should mention, that what I present here is considered very wrong, in particular because it involves hacking a Drupal core PHP file. But after trying to reach an elegant solution and asking for help, I gave up for something that is ugly but works.

This is the ugly part: modify modules/system/html.tpl.php so that the part saying

<title><?php print $head_title; ?></title>

becomes

<title><?php if (isset($GLOBALS['head_title_override'])) { print $GLOBALS['head_title_override'] ; } else { print $head_title; } ?></title>

While I am ashamed of hacking something under “modules/system”, the usage of PHP global variables may be adequate here, even though I’m sure there are more Drupalish ways to accomplish the same. The advantage of a global variable is that it’s easy to set from any execution context.

Having installed and enabled the “Views PHP” module, there’s now a Global: PHP field. It allows access to the data of fields previously fetched, so having the “title” field already in the list, it’s just

$GLOBALS['head_title_override'] = $row->title;

in the “Value code” window. Note that the available variables are listed just below the window. And this should be excluded from display, of course.

Careful with the PHP: If it’s syntactically wrong, the views edit page may be impossible to reach, so the only way to get out is deleting the view. Disabling auto preview may reduce the risk for  this.

Breadcrumbs on Arthemia

For a reason which was beyond me, no breadcrumbs appeared in on my pages. And I ran Arthemia. Until I found the solution here. There’s a bug in Arthemia’s template.php, so

function arthemia_breadcrumb($breadcrumb) {
 if (count($breadcrumb) > 1) {
   return '<div>'. implode(' &rsaquo; ', $breadcrumb) .'</div>';
   }
}

should be replaced with

function arthemia_breadcrumb($breadcrumb) {
 if (count($breadcrumb["breadcrumb"]) > 1) {
  return '<div>'. implode(' &rsaquo; ', $breadcrumb["breadcrumb"]) .'</div>';
  }
}

Looks like the calling convention changed on Drupal 7, but nobody paid attention… This page says a few words about changes in Drupal 7.

Making a local mirror of a site

First, make a copy of the files, of course. Then make a local copy by using a mysqldump backup. Basically, generate the data base with

CREATE DATABASE delme_mirror

from mysql prompt, and then from shell:

mysql -D delme_mirror < databasedump

This can take a few minutes…

Now, allow write-enable on sites/default, make a copy of settings.php within this directory, and make it writable.

Then change the database setting (maybe copy the remote site’s?) so it says something like

$databases = array (
 'default' =>
 array (
 'default' =>
 array (
 'database' => 'delme_xillybus',
 'username' => 'username',
 'password' => 'password',
 'host' => 'localhost',
 'port' => '',
 'driver' => 'mysql',
 'prefix' => '',
 ),
 ),
);

Well, I think this is the thing to do. I didn’t manage to run this, because Drupal went “Fatal error: Undefined class constant ‘MYSQL_ATTR_USE_BUFFERED_QUERY’” which appears to be a PHP version issue (only cutting edge PHP for Drupal, it seems).

Themes

These are my very shallow impressions of a few selected themes I checked up for a hi-tec site. I checked with Firefox 3.6 and IE6 to get a wide range of compatibility issues.

• 0-point: Beautiful and sleek on Firefox, complete disaster on IE6.
• Aqua Marina: Not bad, and survived IE6 test fairly OK, but still pretty bad. Doesn’t have a hi-tec look, though.
• Danblog: Simple & to-the-face design pretty suitable for a hi-tec site, but looks a bit too simple maybe. Survived the IE6 test very well (no differences).
• Danland: Same impression as Danblog. The differences are possibly deeper in.
• Fusion was obviously not intended to be used out of the box.
• Garland looks great, and not so bad on IE6 (resize issues on the latter). Not really a hi-tec look (top region should be bright)

Views

I wrote a separate post on this.

SQL command for strings search in all pages

This is in particular useful for updating links.

It boils down to this:

SELECT entity_id, alias FROM field_data_body LEFT JOIN url_alias ON source=CONCAT('node/', entity_id) WHERE body_value LIKE '%string-to-search%' ORDER BY entity_id;

And then access the page with the URL like e.g. https://thesite.com/node/114 or use the alias as the path instead (if such exists).