Article content
Geometa manages substantial datasets regularly updated as part of business processes. Users input a diverse range of information into the system, often in unstructured formats, presenting a complex challenge that requires specialized knowledge and skills. Take, for instance, urban planning documentation, where a single document may contain data spread across dozens of layers. To simplify these intricate processes, Geometa incorporates special modules and tools, enriching the platform’s capabilities. This article explores the latest advancements, focusing on Geometa’s upgraded Importer Module.
This module is a streamlined integration of two existing solutions — desktop and web-based data import applications. We’ve distilled the best features from both, addressed missing elements, and prioritized key considerations, including: data security requirements, need for fast data delivery, and expandable source file formats.
The module represents an independent service application designed for seamlessly loading data into the system. For users, the process of working with the module is straightforward, encompassing three stages.
Stage 1: Template Selection
In the template, rules are defined for placing source data into the target database. This template facilitates the import of both objects and their relationships. If the desired template is unavailable, users can create one by specifying criteria for searching objects to establish connections, applying filters, and transforming data. The system employs its proprietary query language, designed for maximum simplicity and error-checking during template configuration.
Stage 2: Import Task Creation
Users initiate the data loading process based on the pre-selected template, manually or automatically mapping source files to the types of objects where data should be loaded. Before commencing the import, the system validates the templates using intelligent analysis. The tool evaluates the template structure for consistency between source files and the target database, allowing users to set the import sequence considering the interdependence of the data to be loaded. This saves user time and provides insights into potential issues before the import begins.
Stage 3: Tracking Import Progress
Equipped with a comprehensive log, the module offers detailed information on the import progress and any encountered errors.
This log enables users to:
- Monitor the import task sequence, ensuring predictable import times.
- Retrieve information about previously imported data, including who imported it, when, the number of records in the source file, the template used, and more.
- Identify errors and determine their causes.
Key enhancements in the new version include:
- Enhanced Data Security: The new importer prioritizes data security by eliminating the use of direct database queries. This ensures secure data transfer tools, safeguarding sensitive information during the import process.
- Versatile Plug-In System: A newly implemented plug-in system significantly expands the database of supported data formats. Users now benefit from an extensive range, including popular formats like .gml, .tab, .shp, .gpx, .kml/kmz, .mid/mif, .xlsx, and .csv. Furthermore, the system allows for the seamless addition of new formats, providing adaptability to emerging data standards.
- Efficient Error Management: The new importer incorporates intelligent problem analysis, enabling swift identification and elimination of errors during the creation or import process. This feature streamlines workflows, enhancing user efficiency and reducing the likelihood of data-related issues.
- Optimized Performance with Task Queue Organization: To ensure optimal performance, the import module introduces an organized task queue. This systematic approach enhances efficiency by prioritizing and executing import tasks in a well-structured sequence, contributing to a smoother and more predictable import experience.
- Comprehensive Import Log: The new version introduces a detailed import log, providing users with comprehensive insights into the import process. This log not only facilitates tracking the order of import tasks but also serves as a valuable resource for users to review previously imported data, understand import details, and identify and address any potential issues.
Developing the Module
Creating the enhanced importer module was a multi-faceted process, marked by collaboration, insights from users, and a commitment to innovation. Here’s an overview of the development journey:
In the initial phase, our team engaged in extensive collaboration with users, conducting in-depth work to gather over two hundred user stories. This invaluable input allowed us to define the technical requirements for the new module’s functionality. These insights were then meticulously refined during the prototyping stage.
At the same time, our team engaged in a reverse engineering process on existing products, including the desktop importer and web importer. This effort involved feature analysis, providing crucial insights into user preferences and identifying aspects that were regularly in demand, occasionally used, or deemed unnecessary.
Progressing to the next stage, our team explored existing ETL (Extract, Transform, Load) solutions in the market. This exploration aimed to assimilate best practices and further enhance the functionality and convenience of our future importer module.
One of the major challenges encountered was the imperative need to develop the module without relying on SQL scripts to ensure robust data security. In adherence to modern security requirements, we opted for an alternative approach—using our own grammar to describe import rules. This ongoing endeavor includes plans to introduce new functions, serving as a comprehensive substitute for the traditional SQL-scripts approach, specifically tailored to the tasks users need to solve.
Currently operational in 19 regions, the module is in the crucial phase of processing user feedback. This iterative process allows us to prioritize tasks for further improvement and the continued development of enhanced functionality.
We extend our sincere gratitude to everyone who played a pivotal role in this large-scale project. Your contributions are driving product development and optimizing our collaborative efforts. Thanks to your involvement, the new data import module seamlessly became a part of Geometa’s core suite of software solutions.
