WE DEVELOP TECHNOLOGY FOR NATIONAL HARVESTERS
A Referencia, through its Technical Team, has developed a free software platform (GPL 3) to harvest information that, besides supporting the central node of the regional network, works as an aggregator service and national portal for the member countries. The technological solution is originated from the pilot phase of the IDB project from March to September 2013. In mid-2014, with resources from the Science and Technology agencies, the need for a transferable solution and a set of technological improvements was determined.
Since 2015, LA Referencia technology has been installed in:
- Argentina (MINCYT)
- Brazil (IBICT)
- Colombia (Renata, Colciencias, Ministry of Education)
- Costa Rica (CONARE)
- Chile (CONICYT. LR Harvester y LR Provider)
- Ecuador (CEDIA, SENECYT)
- El Salvador (in installation, CBUES, Ministry of S&T)
- Peru (CONCYTEC)
The platform is composed of three components that work in coordination:
LRHarvester: Harvest component, transformation and validation of OAI-PMH metadata. Responsible for the harvesting of OAI sources, validation of metadata records and their transformation to comply with the quality guidelines of the network. The harvest component was initially based on OCLCHarvester2. The platform is developed in Java 1.7 language, using the Spring framework and the PostgreSQL 9.X database engine. It also works on MySQL.
LRProvider: component responsible for publishing metadata using the OAI-PMH protocol, currently derived from the Dspace-XOAI software, which offers high flexibility and scalability for interoperability with other international metadata aggregators.
Search Engine / Service Portal: aggregated metadata publishing web portal that provides a friendly and adaptive search interface for mobile devices. It also provides statistics on metadata quality and crop evolution. This component is based on the free software Vufind (http://vufind.org) on which specific modules have been developed.
Its latest version (3.0) has improvements such as:
- Responsive and friendlier management interface
- Redesign of the validation and transformation mechanism oriented to the implementation of rules that work with more complex guidelines
- Extending domain objects to reflect and store validators, transformers, controlled values and transformation dictionaries
- Redesign of the relational structure to improve performance
- Storage of more details of motor diagnostics Solr
- Redesign of diagnostic interface and reports
- Multi-rule validation, multi-field and crop improvements to OJS
The design principles defined in 2014 -2015 are still valid:
- Transferable: the version of the central node must be the same as the national nodes.
- Responsive: access by mobile devices (bootstrap)
- Multilingual: support in English and Portuguese
- Business Oriented: focus on metadata validations and transformations in OAI
- Support for OpenAIRE guidelines and flexibility to update new guidelines
- Scalable: You must harvest hundreds of repositories and more than one million records in less than 24 hours.
- Administrable and friendly: diagnostic tools, validation and reports for administrators. Possibility of handling by information specialists and not just computer technicians.
- OpenSource: development and use of components and solutions (eg.: VuFind for the front end or use databases known as MySQL, PostGres, and Java language).
- Statistics: development of new modules
- EEconomical: based on free Linux distributions like Ubuntu and an "average" national node, a two-core machine with 8 GB Ram, 500 GB of space
Software and documentation are available inthe GITHUB repository.