WODA:
The Web Oriented Database
home | docs | features | releases | help | manual | FAQ | download

This document provides information on how to install, create and maintain Web databases using the WODA engine. Database administrators will be doing this. Their username is admin.

Foreword: The are many database systems available and some even let you access the data through the Web. However, most are complicated to install and use. The author of Woda believes that Web services, even simple, personal ones, should be approached as a database application. He created Woda so that making small to medium size database on the Web should be trivial for novices. Experts, however, would still be able to create professional applications. The philosophy behind Woda is that you describe what data your application handles, and all required pages are generated automatically, based on what the data structure is.

In its beginnings, Woda created functionality that was used directly by humans. It was creating HTML Web pages. Recent versions, particularly 4.500 and above also generate Web services functionality so that other Web applications can access data managed by Woda.

Contents:

Technical overview
Downloading and Installing WODA
Creating a database
Running WODA applications
Customizing WODA
Attributes in the definition file		 Relational features
Security and privileges
Maintenance
Contributing to WODA
Compatibility
Where to learn more

1. Downloading and Installing WODA

The author supplies WODA several languages:

This list of languages depends on the availability of the translators. Actual situation may differ. If you would like to help provide WODA for your language, read Section Translating Woda.

The English version should be installed first. The common files, needed in all languages versions, are only a part of the English distribution. Other languages versions should be installed after the English version.

2.1 Required environment

The server onto which you are installing WODA, must let you run Perl CGI scripts. The server administrator should tell you ('directory' means the same as 'folder'):

2.2 Installation procedure

  1. Download WODA. You must pick the English version first.
  2. Unpack the tgz file into a temporary directory, for example /tmp. Woda is archived with tar and compressed with gzip. On Unix you would simply type:

    tar -xvzf the-woda-file.tgz

    On Windows you might need a special program to handle the tar.gz files. Let me suggest the Total Commander. It is the author's favorite piece of shareware. Make sure you do it in such a way that the case of the file names is preserved and directories are created.
  3. Read the readme.txt file.
  4. Run installation program with this command:

    5. perl install

    and answer the questions asked.

  5. Try the installation by opening http://www.yourserver.com/cgi-bin/woda/hello

2.3 Manual installation

The tgz file includes these subdirectories:

2.3 Installation notes for Windows 95, 98 and NT

WODA's natural environment is UNIX but it works on Windows 95 and up. It even works on notebooks without Internet connectivity. In the latter case you should install TCP/IP protocol and use URLs like http://127.0.0.1/ to talk to your server. But you need:

2.4 Troubleshooting the installation

The FAQ document deals with some common problems. They are:

To troubleshoot Woda, access to the httpd server's error log file is essential. It is usually at /var/log/error.

2.5 Securing the installation

After Woda is working, you may want to do the following:

2.6 Some notes on speed

WODA has been found very fast, even in comparison with the SQL databases. What slows it down most is the compile time overhead that Perl incurs with each CGI call. This overhead is lower with Perl 4.036. It is also lower if woda.pl is used, not woda-max.pl. In the first case, Perl just complies what is needed to serve a request (some 100K). In the second case it compiles the entire Woda engine (500K). Woda has been tried with MOD_PERL but the trouble does not seem to be worth the speedup.

2. Technical overview

Now that your demo Woda applications are up and running you may want to learn a few things about Woda to be able to modify them and write your own.

2.1 Data definition language

WODA is a standalone database management system for the World Wide Web. It consists of a Perl program e.g. woda-max.pl (also called the Woda engine) and several support programs (in /usr/local/woda), icons and javascripts (in /document-root/woda/). Each individual database application is defined in the database definition file (definition). This file defines the data dictionary, customizes the user interface and then calls the engine to process the user request and generate the appropriate HTML or XML page. The definition is in two Perl hashes:

One engine therefore supports several databases and is entirely database independent. All operating system dependent parameters are defined within the engine. The engines are language specific. You can help translate Woda into your language.

2.2 Data storage

In WODA terms, a database is in fact a table (but richer than a table in SQL terms). A table can contain an arbitrary number of records. Each record can contain any number of fields. Fields may contain textual or binary information. Fields of type LINK and OPTIONLINK and functions such as &QRY() and &FLD() enable the relating two tables. Also, see a section on relational features.

All data related to a table is stored in one directory, specified in $WBB{dataDir}, which has several subdirectories. System subdirectories have names starting in _underscore. The subdirectories are:

_cache

Stores some system files, search-friendly representation of the data and various files that speed things up. All files but access*.* can be automatically rebuilt.

_data

Stores the data in two files: fat.txt and records.txt.

_doc

Stores html files that are to be post-processed by the Woda engine.

_friend

Stores data related to registered users of the database.

_log

Stores the log files.

_qry

Stores cached queries into other tables.

_tmp

Stores temporary files.

att

Stores binary fields as files.

robots

Stores stuff that we want Web robots to find.

Binary fields such as pictures, sounds and documents are stored each in a separate file named key.fieldName.randomstring.ext where key is the record identification and ext is the original extension of the file such as .gif, .doc, .zip or .tar.gz in the att subdirectory. att/ and robots/ must be in the path that the Web server sees. WBB{htmlDir}, if different from WBB{dataDir}, defines where the att and robots are stored.

The _data directory contains all records of a database in two files, record.txt and fat.txt. Records.txt file contains rows like this:

231 988767
name
Audrey
lastName
Horne
address
Twin Peaks
221 988767
name
Tom
lastName
Peters
address
...

First row tells the size of the record and its modification time in UNIX time format. What follows are attributes and their values. Since items cannot contain newlines, these are escaped into \034 character. Text fields in the database cannot contain the \033 character either. Yes, this may look a bit primitive to database connoisseurs however, it is extremely robust and flexible. Besides, the author is not designing something to compete with Oracle, Sybase or Microsoft in raw database power, but in ease of install, set-up and use! Data is always appended to the end of the record.txt file. A change of the database structure does not cause any problems. This scheme also enables tracking the changes in a record and unlimited number of undo operations.

File fat.txt contains a table with two columns, record id and byte position in the records.txt of that record. This enables faster access to records in the records.txt file. Deleted records are at position -1.

Warning: do not manually edit those two files unless you know very well what you are doing! In particular the records.txt file should not be edited in any mode that allows deletion or insertion of characters. Careful overwriting may be possible.

The efficiency of larger databases, particularly when searching, is vastly improved by caching some information in the _cache directory. Cache files are created from the _data files periodically, either after every update of the database or after every few hours or as a result of a cron job (administrator definable). Several such files will be created, pre-sorted to the defined sort expressions. Some of the files, which are cached (such as those for tree-like tables of contents) depend on the definition files. Changes in the definition files will not automatically update the cache - delete cache manually or touch the fat.txt if rebuilding of the _cache files is needed!

Any file, which starts in an underscore, is WODA system file. Also, system fields start with an underscore. Avoid any names that start with an underscore!

Actual searching is done using brute force using /bin/grep or Perl grep command. This solution has been proven fast enough for databases up to 15 megabytes in size and growing. For heavy load databases the built in caching of search results vastly reduces CPU load.

3. Modifying the provided examples

Woda distribution comes with a number of ready to run applications. They were installed into the /script-alias/woda directory and should be accessible from URL like http://www.yourserver.com/cgi-bin/woda/hello. Some tips before you start hacking those files:

4. Creating a database

Typical steps in creating a database using WODA are as follows (or better still, simply copy and edit one of the demo files you installed):

  1. Create a data definition file in the cgi-directory that has at least  these lines (subject to you installation settings):

    #!/usr/bin/perl
    require '/usr/local/woda/en/woda.pl';
    $WBF{name}='1;';
    do main();
  2. Chmod +x theFile

This will create an application and a Web service that would handle a database with a single field called name. Need more fields? Need more features? There are hundreds of other setting you may wish to define just above do main() row.

3.1 Database creation tips

3.2 Creating a database from sample data

A quick way to create a first draft of the database is to use the "Create from Data" option in the Administration menu:

  1. Prepare the data into a .CSV file using software like MS Access or MS Excel
  2. Fill out the form
  3. Voila, database is ready for first use - if the Web server was allowed to create all required files and directories.
  4. Refine the database using modify structure menu

3.3 Importing data into a database

You can import data from a .CSV file which can be created by software like Excel or Access. The fields in this format are delimited with semicolon or comma. If a field contains new-line, semicolon or a double-quote ("), fields are quoted in double quotes and each double-quote is replaced by two double-quotes. Import only happens if all records pass all controls defined.

4. Running WODA applications

WODA databases can be run:

Typical URL which requests a WODA database service looks like this:

http://www.someone.com/cgi/guestbook/Search?search=ziga&since=12112122

Above, guestbook is the name of the definition file, Search is the action which should be performed, and following the '?' are parameters separated with '&'. A more abstract form of the above example (skipping the server part) is:

/cgi/script[PATH_INFO][?QUERY_STRING]

Braces [] denote optional values. PATH_INFO is what comes after the name of the script (including the slash) and QUERY_STRING are the parameters after the question mark.

Since Version 4.500 the use of Woda through the command line is discouraged and is not documented.

WARNING: Do not use URLs like http://www.someone.com/cgi/guestbook without defining actions. Microsoft Explorer will not correctly pass cookies to URLs. Use http://www.someone.com/cgi/guestbook/Home to show the title page.

4.1 Username and password

PATH_INFO part may also include username and password. The requests would look like:

/cgi/definition[/u=username&p=password][/Action][?QUERY_STRING]

for example:

/cgi/guestbook/u=guest/Search?search=ziga

5. Customizing WODA applications

5.1 The definition file

The definition file is actually a Perl program which defines what the database looks like and then calls the woda.pl (the engine) to handle the database. Database definition consists of two kinds of attributes. The firsts (WBB) define common features of the database the seconds (WBF) the features about each filed in the database. Few of the attributes must be defined, most are optional and can be left undefined. Full documentation on the configurable parameters is generated by Woda and available from the administration menu. There are two ways to define the attributes:

String expressions

String expressions are expressions which result in a Perl string. Three examples of a string expression are:
        "Author name: " . $rec{name}
     "Author name: $rec{name}"
     "<A HREF=$this$rec{title}</A"

The first line uses the dot operator to concatenate strings. The second and third one make it a string by putting it all inside double quotes. See below for explanation on the variables ($rec, $this) you can use.

Logical expressions

A logical expression is anything that can be interpreted as a true or false value. 0 or empty strings mean FALSE, nonzero and non-empty strings mean 1. Examples: 

        $_ == 20
        $_ eq "USD"
        m/.{5,20}/

Is the value numerically equal to 20; is it equal to "USD" as a string? Is it between 5 and 20 characters long. This last is in fact a regular expression.

Regular expressions

Regular expressions match a string against a pattern. E.g.: 

        \.com$
        .*
        ^193\.2\.92\..*

First matches strings which end in ".com" (note the backslash which makes sure the dot is not interpreted as a wildcard. Second matches any number of repetitions (star) of any character (dot). Third matches strings which start with 193.2.92. Note that . means anything and * any number of repetition of the previous character, ^ means the beginning and $ the end.

Predefined variables and functions

In the above expressions variables may be used. The predefined values are:

You can use any Perl function or expression. See next section for some more functions. See "Display Global Variables" item in the Admin menu to see what global variables WODA uses and which you could use in expressions. Do not set them, just read them.

5.2 Internationalization

Woda does not (yet) use UNICODE. A set of $WBB{intl*} attributes allows administrator to define:

In addition, the WODA GUI can be translated.

6. Relational features - linking tables

Relational features in Woda work on the same server or across several servers using HTTP. This is what you can do:

Most of these function expect a reference to another table as a parameter. If the application is cgi-bin/people with data in somewhere/people and the other application is cgi-bin/other with data in somewhere/data you do not need to do anything. Just call the other table 'other'. If, however, the other table is somewhere else on the same server or even on another server somewhere on the Internet, then a setting $WBB{tables} must be defined. More information there.

6.1 Example of setting up a relation between two tables

Suppose we have two tables - a table of professors and a table of classes. Each professor teaches 0 or more classes. In the definition of the classes table, we would include a field that would point to the class's professor like this:

# in cgi/classes

$x='professor'; # ------------------------------
$WBF{$x,srt}=$i--;
$WBF{$x,'p'} = 'Professor teaching this class';
$WBF{$x} = '1;';
$WBF{$x,'type'} = 'LINKOPTION';
$WBF{$x,'into'} = 'professors';
$WBF{$x,'picture'} = '"<A HREF=/cgi/professors?$_>$_</A"';

in the above example only the id code of the professors record will appear in the pulldown. If we modify the above into field as:

$WBF{$x,'into'} = '&ROWS("professors","_id,firstName,lastName")';

the pulldown will include first and last name of the professor.

If we further improve the picture field as:

$WBF{$x,'picture'} = '
$xx = &FLD(professors,$_,lastName) . " " .&FLD(professors,$_,firstName;
"<A HREF=/cgi/professors?$_>$xx</A"
';

the fist and last name of the professor will also be nicely printed in the tabular output form.

Finally, in the professors record, we might like to see a list of classes he/she teaches:

# in cgi/professors
#
$x='classes'; # ------------------------------
$WBF{$x,srt}=$i--;
$WBF{$x} = '1;';
$WBF{$x,'type'} = 'COMPUTE';
$WBF{$x,'picture'} = '"<A HREF=http://fgg.uni-lj.si/cgi/classes/Search?search=$rec{_id}>
Classes he/she teaches</A"';

The above only displays a link to a query. The below modification displays the actual records:

$WBF{$x,'picture'} = '&QRY("classes","+professor:$rec{_id}");';

The above approach may not be too smooth, but it works. The &RELATED, &DECODE and &LINK functions make this even simpler.

7. Security and Privileges

WODA distinguishes between users and groups of users. Access to individual actions within WODA is still based on a group to which a user belongs to. Three levels of security can be assigned to a WODA table:

  1. Minimal; groups only. No groups are defined by the administrator. WODA creates three groups: admin, friend and guest. Admin can do anything. Guest can do anything except access the admin actions or order agent searches. Friends are guest that chose to register.
  2. Custom; groups only. Other groups are defined in $WBB{groups} and access rights of the groups are defined in $WBB{rights}.
  3. Custom; groups and users. A separate WODA table for user information is set up. $WBB{userTable} should be set to point to it. Other $WBB{user*} settings are used to define to which group all these users belong (unless their group is defined in their $rec{group} and on how to evaluate if the user is allowed to use the database.

7.1 Defining groups

Groups are defined in $WBB{groups} field. This field contains a table with fields: groupname, IP address and password: 

        joe             193.8.9.34     password1
        company         we.com$        password2

WODA starts at the top of the table and looks for a match of address and password (if defined). When a match with the supplied information is found this is the group. There are two default users, admin and guest. Admin's information is part of the WODA setup!

7.2 Passing group (or user) and password information to WODA

There are three ways to do this:

  1. By IP address of the client workstation. It will be matched against second column above.
  2. By passing username and password via URL. See above how to do this.
  3. By logging in using the login button in the footer of all WODA pages and passing this information later via a cookie.

WODA first tries to interpret the username as username and match it against the users in the $WBB{userTable} database. If this fails, the username is interpreted as groupname. Beware that usernames override the groups, except groups guest and admin!

7.3 Allowing/denying actions

Users may be allowed or denied actions in the database. Actions specified in the URL before the ? and after the script name. Actions may be allowed or denied for some users in a table defined in $WBB{rights}. This field is a table like: 

        allow   joe     ^Search
        deny    .*      ^Search

In the first column is the word allow or deny. In the second is a pattern which will be matched against current user's group and in the last the pattern matched against the requested action (as shown on the URL). The program will start at top and process lines as long as it does not find a match in both user and action. The it will either allow or deny it. If there were no matches, the action is allowed. Above only joe and admin (of course) will be allowed to search.

Read only database would have rights like:

$WBB{'rights'} = 'deny .* ^Edit|^Delete|^Add|^Adm';

Note that if the rights field is defined, Adm* actions have to be disabled as well. The actions that are not allowed will not show in menus.

7.4 Allowing/denying access to the field

In the $WBF{field,sees} a pattern of users is defined, which can see the value of the field. In $WBF{field,modifies} a pattern of users is defined which can change the value of the field. If undefined, anyone has access.

Warning: Users are still able to search by fields they do not see. They will not be seen in WODA generated printouts, but will be displayed in the $WBB{format;DEFAULT} expressions if explicitly required.

7.5 Denying access to records

Generally, every record entered into a WODA database is immediately available to all users of the database and found in all searches. Administrators can set up a database in such a way that they clear the record before it is available to others or that they can simply filter out some records which contain e.g. obscene content.

WBB{filter} defines and expression which is silently appended to every query string. So that if you put there -sex, no records which contain word sex will be found in searches. If you set

$WBB{filter} = "+qwerty"

only records which include string qwerty will be found. Administrator should use 'sees' and 'edits' field options so that only he/she can put values into a field which only he/she sees and is allowed to edit.

8. Maintenance

The admin should, periodically, clean the wdat/*/_tmp directory and remove archival versions in the _log and _data directories. Note that there is a primitive Web based file manager built into Woda.

9. Compatibility

9.1 IIS

In a few important ways Microsoft's Internet Information Server does not behave as most decent Web servers do. In particular it does not set the PATH_INFO environment variable correctly and does not parse the PATH_INFO correctly. For example, given a URL like: /cgi/myprog/Search it will look for program Search in the directory /cgi/myprog. While in fact it should execute myprog and pass it /Search as a PATH_INFO. It took a lot of work to rewrite WODA to go around this.

9.2 Windows

A lot depends on a particular port of Perl and Web server that you are using. The author finds the standard Perl + Xitami 100% compatible with the exception of the passwords with which the records are locked. This means that records encrypted on a PC would not be decrypted on UNIX properly.

11.3 Mod_perl and woda*.pm

Support for Modperl is discontinued in Version 4. It may work, but the author is not using it.

To use WODA with mod_perl you should uncomment the package row in woda*.pm.

In this case WODA code is compiled into package WODAUK. All processing is done within that package, as well as are the evaluations of all strings defined in the database definition file. If you defined functions or constants in the definition file you should also do this.

10. Contributing to the Woda project

There are several ways to customize databases that use WODA engine.

  1. By overloading functions defined in the WODA engine with custom functions (for experts).
  2. By adding new functions to Woda (for experts).
  3. By translating Woda user interface.

It would be great if the modifications and translations could be shared among Woda users!

10.1 Overloading built in functions

If a user requests page like

/cgi-bin/people/Show?search=smith&sort=TIME

this request will be handled by a function cgiShow. WODA searches for this function in the following places in the order as they are listed:

It is expected that your modification or extensions will be in the cgi-bin/people, if they are local to the people application, or in the local/cgiShow.pl, if they will be used in several applications on that server.

Each cgiFunction get the parameters passed to it with HTTP POST or GET request in the has $CGI in the case above like:

$CGI{search}=smith
$CGI{sort}=TIME

in the hash ENV is the HTTP and OS environments. Redefined cgiShow function should look like this:

sub cgiShow {
	&printHead("Header to print on top");
	# whatever you want it to do comes here
	&printFoot();
}

The function may use any other woda function defined in the woda.pl file. Brief documentation on the WODA programmer's API is available from the Admin's menu.

10.2 Extending Woda with new functions

Using the mechanism described above one can create a totally new function. After it is tested it can be submitted to the author of Woda for inclusion into the contrib/ section of the distribution.

10.3 Translating WODA

All strings from WODA, except from the administration sections, are extracted into a translation file. The format of the file is quite simple: 

==========
Original lines in the
English Language
----------
N O T   T R A N S L A T E D:
Original lines in the
English Language (which need translation).
==========
Original lines in English
----------
Translations
==========

To translate, one has to:

  1. Translate the text below the NOT TRANSLATED line and above =======
  2. Delete the NOT TRANSLATED line.

Some translation rules:

You can download sample translation files from the "files" directory.

Home of the Italian and German translations of WODA.

11. Where to learn more

Follow the links from the documentation section of the Administration menu of WODA.

There is a help file which is pointed to from the WODA applications, which has some more information on creating and managing WODAs. Release notes provide information about new features.

 WODA: the database tool for the World Wide Web
March 3, 2009