The Ukiyoe CMS


Advanced

Ukiyoe has a built-in search engine which can handle simple queries, and its Unicode support extends to URLs and paths. In addition to the public area, ukiyoe enforces encrypted connections to the private area, which can be further protected using Apache's authentication mechanism.

Renaming public and private directories

By default ukiyoe serves articles non-encrypted from public, and uses HTTPS for articles in private. Changing these names highlights some of ukiyoe's features and pitfalls.

The following uconfig.py modifies this behaviour:

# -*- coding: utf-8 -*-
CONFIG = {}
CONFIG['public_area'] = 'home'
CONFIG['private_area'] = 'secure'
CONFIG['front_area'] = CONFIG['public_area']
CONFIG['area_re'] = '(%s|%s)' % (CONFIG['public_area'], \
                                 CONFIG['private_area'])
CONFIG['uki_dict'] = {'{/+}': '/%s/' % CONFIG['public_area'], \
                      '{/-}': '/%s/' % CONFIG['private_area']}

The "public" area is now called home, while HTTPS is enforced on secure (the corresponding change must also be done in the filesystem, of course). Note that the areas must be the name of the directories, and that no subdirectories are allowed. Even though the definition of CONFIG['area_re'] hasn't changed, it must be set again since it depends on the redefined values.

CONFIG['uki_dict'] is a replacement table which is applied to the metadata and content before the Markdown filter (if any) is used. It is superceded by the CONFIG['uri_tag'] replacement (see Settings, {/} by default), but can otherwise be used as a shorthand for certain strings in the uki data files. In this case {/+} will be replaced by the value of /CONFIG['public_area']/ (i.e. /home/), while {/-} will be replaced by the value of /CONFIG['private_area']/ (i.e. /secure/). Thus, any subsequent changes of those parameters will automatically propagate throughout the CMS if links are specified using the shortcuts (e.g. {/+}article-name).

Searching

Ukiyoe has a search engine which can be embedded in a template by means of the following code:

<form action="/{{config['search_route']}}" method="post">
  <input name="search_query" type="search" />
  <input class="button" value="Search" type="submit" />
</form>

The search is a case-insensitive "and" query over metadata title, keywords and article content. It is neither fast nor versatile, and it cannot handle languages which do not separate words with spaces (but you can try, for example, the word "español" below):

Note that in an article (that is, a .uki file) the search route will not be replaced by the template engine, and hence the route has to be hard-coded i.e. action="/search" instead of action="/{{config['search_route']}}".

Having a good set of keywords in the metadata increases searching speed considerably (while decreasing memory usage). If keywords suffice it may be worthwhile to set CONFIG['search_keywords_only'] = True in uconfig.py (see Settings). If search is causing performance issues it can be disabled by setting CONFIG['enable_search'] = False. A large performance gain can be obtained in exchange of some accuracy by setting CONFIG['search_sans_html'] = False, which does not strip HTML code from the content (minimal to begin with if Markdown is used).

In addition to CONFIG, the search template is provided with the searchwords set containing all the search terms, and the matches dictionary which has the same structure as the index dictionary (non-parsed content) but which only lists the articles which matched the search query, and the client's IP address via client_ip. A query dictionary contains all other parameters passed on via POST (GET is not supported), including the raw search_query.

It is important to remember that any search words or query values displayed in the template should be escaped, i.e. write {{word}} instead of {{!word}}. Furthermore, care should be taken to avoid information leaks, as private and restricted articles are indexed (but can be filtered in the template).

Note that the search template cannot index, i.e. a search.meta file will be ignored.

Uploading a file

By default, file uploading is not allowed until CONFIG['enable_uploads'] is set to True. The upload URI (the web path where upload.uki will reside) can be set via CONFIG['upload_area'] and CONFIG['upload_item']. It's important to note that ukiyoe only provides the upload mechanism, the actual file creation and storage is delegated to the template (but if using the provided template do revise CONFIG['upload_path']). In addition to the config, metadata (from upload.uki), and query dictionaries, the following are provided to the upload template:

and finally upload, a bottle FileUpload instance.

An example of an upload page can be found here. Before using it make sure to read the notes at the top of upload.uki, which also provides other ways to control how uploading is performed. Note that the upload template cannot index, i.e. an upload.meta file will be ignored.

Backup, recovery and filesystem permissions

An entire ukiyoe site can be backed up at any given time (e.g. 12dec2013) as follows:

zip -r -9 ukiyoe.12dec2013.zip ukiyoe/

The site can be recovered from backups as so:

unzip ukiyoe.12dec2013.zip

Any other archiving scheme (e.g. tar) is, of course, acceptable. Individual areas and/or items may be backed up the same way. Furthermore, the underlying filesystem permissions and attributes may be leveraged, for example, by changing directory permissions or the immutability bit.

Error pages

Error messages are handled by the template defined by CONFIG['error_tpl']. The template is provided, in addition to the CONFIG dictionary, the values of code (the numeric error code), and details about the error in details.

Unicode support

Ukiyoe should be able to handle non-English posts, searches, and paths. By default item and file names are restricted to the following RE (regular expression) patterns:

CONFIG['dir_re'] = '[^/\.]{1,80}'
CONFIG['file_re'] = '[^\.]{1,80}\.[^\.]{0,3}[^~]'

Item directories cannot contain a forwards slash or a dot, and files cannot start with a dot, must contain one dot (for the file extension), and cannot end with ~ (so as to ignore Emacs backup files). The 80 character limit is arbitrary, but needing longer names seems unlikely. They can, furthermore, be made even narrower in uconfig.py (for example, CONFIG['dir_re'] could be [A-Za-z0-9_+\-]{1,40} and CONFIG['file_re'] set to '%s\.[A-Za-z0-9]{1,4}' % (CONFIG['dir_re'],) to enforce an ASCII URI, plus some naming restrictions). Again, file names must have a dot in order to distinguish them from items which should not have one.

Regardless of how they are set, the following is enforced in-code for both item and file names: cannot start with a dot, contain a filesystem path separator, or have sequential dots anywhere.

Consider the following uconfig.py in ukiyoe/:

# -*- coding: utf-8 -*- 
CONFIG = {}
CONFIG['rss_title'] = "浮世絵 コンテンツマネージメントシステム RSS"
CONFIG['rss_description'] = "A feed in Japanese (日本語)"

and then creating an item (note the spaces, which are also allowed by the above RE):

mkdir ukiyoe/public/"浮世絵 test article"

with a file 基本.uki containing the following:

release: 23:00/16/dec/2013
title: A test in 日本語
かぎ: "かぎ" means "key" in English
<-- uki -->
Testing unicode support in 浮世絵.
===
This article uses the template "基本.tpl" which is identical to "basic.tpl"
except for the name. Here is a search term: 小倉奈々.

Here is a file with a non-English name: <a href="{/}Файл.txt">Файл.txt</a>

and, finally, an auxiliary file:

echo 'привет мир!' >  ukiyoe/public/"浮世絵 test article"/Файл.txt

You can now access the article at: http://localhost:8080/page/浮世絵 test article/, and see the RSS entry at http://localhost:8080/rss. Searching for 小倉奈々 should return two results.

The 基本 template does not explicity use the かぎ key. The reason for this is that its inclusion depends of the version of Python being used. For version 2.X, the following code will retrieve the key value (say, just before the {{!content}}) line:

Metadata keys can also be in other languages (Python 2.X): {{metadata['かぎ'.decode('utf-8')]}}

Version 3.X, however, simply requires:

Metadata keys can also be in other languages (Python 3.X): {{metadata['かぎ']}}

Internationalized domain names have not been tested. It's possible they will work only under 3.X.

Note that although articles can have non-ASCII names, Bottle may have issues with non-ASCII directories anywhere along the path to system files and templates (this seems to be fixed in Bottle 0.12). If using non-ASCII file names it's better to run ukiyoe under Python 3.X, as some pathological cases may have issues with Python 2.X (e.g. setting a non-ASCII file name extension in lieu of .uki).

When using the lynx browser make sure to set the Display character set to UNICODE (UTF-8).

Configuring Apache

If you've followed the tutorial thus far it'd be best to start from a clean slate by unzipping the original download in a new directory to start over.

So far ukiyoe has been running on localhost:8080 using the bottle HTTP server. To replace it with an Apache server running mod_wsgi the first thing to do is modify the ukiyoe.wsgi file (inside ukiyoe/) to specify the full path in which the ukiyoe root directory resides. For example, if the ukiyoe directory is transferred into /www/wsgi/ it would be necessary to replace the line sys.path = ['/ukiyoe/'] + sys.path in ukiyoe.wsgi with sys.path = ['/www/wsgi/ukiyoe/'] + sys.path so that ukiyoe.wsgi now reads:

import os, sys
sys.stdout = sys.stderr
sys.path = ['/www/wsgi/ukiyoe/'] + sys.path
import bottle
import ukiyoe
os.chdir(os.path.dirname(__file__))
application = bottle.default_app()

Apache versions and configurations vary considerably, so use the following as a guideline only. In this scenario Apache runs as user apache in group apache. Permissions are pre-configured in the zip file, but can be set as follows:

zsh -f
chmod 555 /www /www/wsgi /www/wsgi/ukiyoe
cd /www/wsgi
chmod 500 ukiyoe/**/*(/)
chmod 755 ukiyoe/private
chmod 700 ukiyoe/public
chmod 400 ukiyoe/**/*(.)
chmod 500 ukiyoe/ukiyoe.py
chmod 444 ukiyoe/private/.htpasswd
chmod 770 ukiyoe/log
chown -R apache:apache ukiyoe

Note that, for extra security, ukiyoe should be run outside the apache DocumentRoot.

The following configuration is a complex example which shows how to set up both an ukiyoe site and a maiko forum in a single virtual host over ports 80 (ukiyoe unencrypted), 8080 (maiko unencrypted), and 443 (ukiyoe encrypted). Care should be taken to make sure the directives below do not clash with the ones in httpd.conf and ssl.conf. A local MathJax installation is also contemplated.

Note that obtaining a signed SSL certificate is beyond the scope of this tutorial, but if one is needed the following recipe can be attempted in order to obtain it from Let's Encrypt (replace example.com with the appropriate domain name):

systemctl stop httpd
systemctl stop iptables
letsencrypt certonly --standalone -d www.example.com -d example.com
chmod 600 /etc/letsencrypt/archive/www.example.com/*
cd /etc/pki/tls/private
ln -s /etc/letsencrypt/live/www.example.com/privkey.pem https.key
cd ../certs
ln -s /etc/letsencrypt/live/www.example.com/cert.pem https.crt
ln -s /etc/letsencrypt/live/www.example.com/fullchain.pem server-chain.crt
systemctl start iptables
systemctl start httpd

Assuming the server name is www.example.com (IP address 93.184.216.119), the following code can be saved in an Apache configuration file (say, server.conf, which would reside in the conf.d directory) in order to create an ukiyoe/maiko virtual host server (tested with Apache 2.4.6):

Listen 93.184.216.119:80
Listen 93.184.216.119:8080
ServerName www.example.com
ServerAdmin webmaster@example.com
<Directory "/www/wsgi">
    Options None
    AllowOverride None
    Require all granted
</Directory>
WSGISocketPrefix run/wsgi
WSGIProcessGroup ukiyoe
WSGIDaemonProcess ukiyoe user=apache group=apache
WSGIScriptAlias / /www/wsgi/ukiyoe/ukiyoe.wsgi
<IfModule log_config_module>
    LogFormat "%h %u %t %D %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" extended
</IfModule>
<VirtualHost 93.184.216.119:80>
    ErrorLog logs/ukiyoe_error.log
    CustomLog logs/ukiyoe_extended.log extended
    Alias /MathJax /www/wsgi/MathJax
</VirtualHost>
<VirtualHost 93.184.216.119:8080>
    WSGIProcessGroup maiko
    WSGIDaemonProcess maiko user=apache group=apache
    WSGIScriptAlias / /www/wsgi/maiko/maiko.wsgi
    ErrorLog logs/maiko_error.log
    CustomLog logs/maiko_extended.log extended
</VirtualHost>
<VirtualHost 93.184.216.119:443>
    LogLevel warn
    SSLEngine on
    ErrorLog logs/ukiyoe_error.log
    CustomLog logs/ukiyoe_extended.log extended
    SSLProtocol all -SSLv2 -SSLv3
    SSLHonorCipherOrder On
    SSLCipherSuite **see note below**
    SSLCertificateFile /etc/pki/tls/certs/https.crt
    SSLCertificateKeyFile /etc/pki/tls/private/https.key
    SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt
    BrowserMatch "MSIE [2-5]" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
    CustomLog logs/ssl_ukiyoe_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
    <LocationMatch "^/private">
        AuthType Basic
        AuthName "Authentication Required"
        AuthUserFile "/www/wsgi/ukiyoe/private/.htpasswd"
        Require valid-user
    </LocationMatch>
</VirtualHost>

Regarding the SSLCipherSuite, please refer to the Mozilla Wiki and SSL LABS.

Note that due to the fact that ukiyoe is serving the files we need to use LocationMatch instead of Directory for authentication.

Stopping the bottle server on port 8080 with Ctrl-C and starting Apache should now provide ukiyoe access at http(s)://www.example.com and maiko at http://www.example.com:8080. See the Apache documentation and the mod_wsgi documentation for details. Keep in mind that although content is instantly updated, template files are cached, and therefore a touch ukiyoe.wsgi should be done to update the cache (if WSGIScriptReloading has not been disabled).

Ukiyoe only allows encrypted connections to the private area (as configured by CONFIG['private_area'] in uconfig.py). The default directory is called private, and access to it can be controlled through the file .htpasswd therein. To populate the file the htpasswd or htpasswd2 commands are needed, available via yum install httpd-tools on Red Hat/CentOS or zypper install apache2-utils on OpenSUSE (and presumably through the package manager of any other mainstream Linux/Unix-type distribution). Once installed, users and password can be added to .htpasswd using the following commands:

htpasswd -c private/.htpasswd tsubomi

or

htpasswd2 -c -B -C15 private/.htpasswd tsubomi

where tsubomi is the name of the user to be added. This creates an entry of the form:

tsubomi:$2y$15$2ltRbq65Zehye.lnMIaNWuUXRx8waMa.rLHO8nSQA5zAUR3zwYITO

Any number of users can be added thereof (without the -c). A sample article can be found at http://<yourdomain>/private/secret (which will redirect to an encrypted connection and require the tsubomi password from above). In the htpasswd and htpasswd2 files included with ukiyoe the example user and password are one and the same: tsubomi and tsubomi.

Access to the private area is always encrypted, including the transmitted password and any files contained within. RSS feeds, searches and indexing can also be controlled via CONFIG['rss_private'], CONFIG['index_private'], and CONFIG['search_private'], although in the case of the latter two information is held in a separate dictionary key from the public area, allowing for selective disclosure (but care should be taken when allowing RSS feeds or divulging site access statistics). Of course, the release control works in the same manner as before.

Adding a comments section

Maiko can readily provide ukiyoe articles with a comment system. Please refer to this page for details.

Regarding security

Ukiyoe has no concept of cookies, sessions or accounts. Normally the administrator (i.e. whoever set up ukiyoe in the first place) would also be the content provider who uploads their own files through an encrypted channel (e.g. sftp). It is dangerous to upload untrusted templates as these are essentially Python programs. It's also unsafe to upload untrusted articles due to XSS attacks. Access to the private area can be given on the basis of hashes (never passwords) provided by users who run the appropriate htpasswd command (again, sent through encrypted channels).

Site visitors should have no real access to the underlying CMS, and ukiyoe does its best to make sure the few methods of untrusted input are scrutinized for suspicious data. Paranoid admins might want to take extra steps such as making all static files and directories immutable, disabling statistics, searches, RSS, and narrowing the file and directory regular expressions previously discussed (however, the harshness of some of those measures would probably not be worthwhile in exchange of little extra security).

Updating ukiyoe

Ukiyoe can be updated by performing the following steps:

  1. Back up bottle.py and ukiyoe.py
  2. Stop the web server
  3. Over-write bottle.py and ukiyoe.py with the new versions
  4. Start the web server

Previous: Tutorial

Next: Settings