India Environment Portal

Built on Drupal, the India Environment portal has over 300,000 nodes, made searchable using the powerful Apache Solr search

Highlights:

• Over 300,000 articles in database
• Migration of data (with massive data transformation and clean-up) from 3 different database sources
• Migration scripts for ongoing data migrations from MS-SQL to Drupal (MySQL)
• Over 8000 tags as part of the Taxonomy
• One of the largest installations of the powerful Apache Solr search for free text searching and tag-based (author and source as well) filtering of articles

URL: www.indiaenvironmentportal.org.in

Background

Centre for Science and Environment (CSE) were inspired to build an environment portal as part of the push for portal development from the National Knowledge Commission, Govt. of India. CSE decided to release 20 years of their research articles spread over 3 proprietary systems, including a Library Management System, a Content Management System written in ASP/MS-SQL for managing the website of their premier Science and Environment magazine - Down to Earth (DTE) - and some Access Databases. The total number of records exceeded 300,000.

CSE also had a very extensive tag vocabulary, or thesaurus, of their library tags, organized as What, Where and Who lists, which represented nature of articles, geographies they represented, and people they were associated with including the authors, respectively. All the articles in its library and DTE database had been classified tagged with these.

Srijan’s initial engagement

1. CSE’s vision of the portal was still evolving, and being a non-profit they had started out with a very limited budget
2. Srijan was engaged with a small Information Architecture exercise for the portal, which included studying of their library management system, the way they tagged articles, and gain an understanding of DTE

Challenges

The DTE content management system had been around for 4-5 years and had grown along the way to meet the changing requirements of the print magazine, without any documentation whatsoever. The database therefore was in a bad shape from a design perspective, with redundant tables and data, adding to the confusion.

Solution

Srijan proposed to break the project into multiple phases, with the first being a pilot for data migration of the DTE database, which formed the bulk of the data, into an open source Content Management System.

Choice of platform

The initial choices were Drupal or TYPO3. We eventually chose Drupal, for two specific reasons namely:

1. Drupal had excellent vocabulary and tag management core modules, which would have to be written in TYPO3
2. TYPO3 had a separate Admin interface which would have proven to be difficult to manage and use for the library team
3. Drupal’s caching mechanism is better than TYPO3; this would be a critical requirement in the high volume traffic that CSE was expecting for the portal
4. A powerful and fast search engine was required for searching through 300,000 articles. Drupal had an Apache Solr search module integrated with it, which was a candidate for implementing this search . 

Key challenges during migration

1. Looking at the MS-SQL database we knew that the database architecture was poor with data redundancies.
2. There were some cases where UNIQUE fields were having duplicate values!
3. Database Normalization was missing. Thus the flexibility, data integrity and  efficiency were not so good. Required to safeguard the database from anomalies
4. Since the data was populated through a Library Management tool, the data were having many special characters which were stored in database. During migration of those data to Mysql database, we had to clean up those. It was very much time consuming and repeated work to cleanup the data without the availability of any documentation of the database
5. Designing our database(MySQL) to accommodate the specific concepts of box stories and cover stories that comes with a magazine like website. It required, understanding the relationships of tables and data, the way it was maintained in MS-SQL database. Again, many redundancies/anomalies in the database for these tables.

Finding a powerful and easy way to search

The next phases of the project comprised of setting up Drupal (with a default theme) and build all the required modules to start showcasing the content in the desired manner. A key component of these phases was the setting up of a powerful search.

Apache Solr was researched with, and selected as the choice of the search. It turned out to be an excellent decision, as in retrospect, not even Google Mini (the other search candidate) would have proved to be so beneficial for researchers. In another case study specifically on Apache Solr, we will describe how researchers within CSE are using the same to their absolute delight.