Geometa’s Enhanced Importer Module

Geometa manages substantial datasets regularly updated as part of business processes. Users input a diverse range of information into the system, often in unstructured formats, presenting a complex challenge that requires specialized knowledge and skills. Take, for instance, urban planning documentation, where a single document may contain data spread across dozens of layers. To simplify these intricate processes, Geometa incorporates special modules and tools, enriching the platform’s capabilities. This article explores the latest advancements, focusing on Geometa’s upgraded Importer Module.

This module is a streamlined integration of two existing solutions — desktop and web-based data import applications. We’ve distilled the best features from both, addressed missing elements, and prioritized key considerations, including: data security requirements, need for fast data delivery, and expandable source file formats.

The module represents an independent service application designed for seamlessly loading data into the system. For users, the process of working with the module is straightforward, encompassing three stages.

Stage 1: Template Selection

In the template, rules are defined for placing source data into the target database. This template facilitates the import of both objects and their relationships. If the desired template is unavailable, users can create one by specifying criteria for searching objects to establish connections, applying filters, and transforming data. The system employs its proprietary query language, designed for maximum simplicity and error-checking during template configuration.

Stage 2: Import Task Creation

Users initiate the data loading process based on the pre-selected template, manually or automatically mapping source files to the types of objects where data should be loaded. Before commencing the import, the system validates the templates using intelligent analysis. The tool evaluates the template structure for consistency between source files and the target database, allowing users to set the import sequence considering the interdependence of the data to be loaded. This saves user time and provides insights into potential issues before the import begins.

Stage 3: Tracking Import Progress

Equipped with a comprehensive log, the module offers detailed information on the import progress and any encountered errors.

This log enables users to:

Key enhancements in the new version include:

Developing the Module

Creating the enhanced importer module was a multi-faceted process, marked by collaboration, insights from users, and a commitment to innovation. Here’s an overview of the development journey:

In the initial phase, our team engaged in extensive collaboration with users, conducting in-depth work to gather over two hundred user stories. This invaluable input allowed us to define the technical requirements for the new module’s functionality. These insights were then meticulously refined during the prototyping stage.

At the same time, our team engaged in a reverse engineering process on existing products, including the desktop importer and web importer. This effort involved feature analysis, providing crucial insights into user preferences and identifying aspects that were regularly in demand, occasionally used, or deemed unnecessary.

Progressing to the next stage, our team explored existing ETL (Extract, Transform, Load) solutions in the market. This exploration aimed to assimilate best practices and further enhance the functionality and convenience of our future importer module.

One of the major challenges encountered was the imperative need to develop the module without relying on SQL scripts to ensure robust data security. In adherence to modern security requirements, we opted for an alternative approach—using our own grammar to describe import rules. This ongoing endeavor includes plans to introduce new functions, serving as a comprehensive substitute for the traditional SQL-scripts approach, specifically tailored to the tasks users need to solve.

Currently operational in 19 regions, the module is in the crucial phase of processing user feedback. This iterative process allows us to prioritize tasks for further improvement and the continued development of enhanced functionality.

We extend our sincere gratitude to everyone who played a pivotal role in this large-scale project. Your contributions are driving product development and optimizing our collaborative efforts. Thanks to your involvement, the new data import module seamlessly became a part of Geometa’s core suite of software solutions.

Interested In How Geometa Can
Help Your City?

Backed with Geometa, we turn cities into digital ecosystems, provide smart city management, drive digital urban development, ensuring transparency, superior quality of life, and economic growth. Let our team walk you through a personal demo!