Datafordeleren - public data redeemed and securely published in one single model
A case study of the work that NorthTech carried out as subcontractor for Datafordeleren
NorthTech has been involved in geospatial data projects for many years. First in the work with Kortforsyningen, which was the first Danish solution for a user-protected portal for public maps on the web. As a result of the increased digitisation and the need for public access to many basic data, in 2012 the political decision followed to redeem public data for common and free use (Digitisation strategy for 2011-2015). This resulted in the project Datafordeleren. The project was put out to tender by the Agency for Digitisation, and won by KMD A/S. From the start, NorthTech was deeply involved as a subcontractor in the implementation of the Datafordeler. Due to the fact that we were experienced with IT architecture, with online user restricted map distribution, and last but not least as a result of our geo-spatial expertise.
Kortforsyningen, which was run and developed by The National Mapping and Cadastral Agency (now Agency for Data Supply and Efficiency), was overwhelmed by its success. For this reason a need for a larger and more modern national platform occurred: Datafordeleren. Most importantly the geodata distribution had to be modernized and streamlined. In addition to this, the agency wanted it to contain more data than just map-based geo-spatial data such as maps from eg. FOT (GEO Denmark). The Datafordeler also had to handle data such as company information from the Central Business Register (CVR-data), housing data from the Building and Housing Register (BBR-data) and data from the Danish Property Register (ESR). The work was launched under the influence of the European INSPIRE Directive, whereby national basic data also should be made available for other EU countries. Finally, the government wanted to create growth opportunities for the public authorities, entrepreneurs, citizens and private companies through the use of free basic data.
About the project
With 4 employees NorthTech had huge influence in the project team of 50 men, that had been selected by KMD. Our good partner, Septima (also a geo-spatial data specialized developer house) participated in this team as well.
NorthTech assisted in choice and processing of the IT architecture, the user management, the platform selection, and the programming of software for the geo-distribution itself, targeting high availability and compliance. We took part in the selection of interfaces (API), we optimized data sets, and we we executed the testing of the product.
The basic data model
NorthTech is a company that takes pride in focusing on the good product rather than on a special technology or method. For this reason we were very pleased to programme the Datafordeler in C #. However, as Microsoft's SQL server was suggested to handle the geo-spatial data, NorthTech and Septima instead recommended a PostGIS server, since it has become the de facto industry standard. KMD chose to follow this advice, and for this reason the solution was optimized in terms of speed.
The Datafordeler is built on the OGC geo-spatial standards (Open Geospatial Consortium), bridging ISO standards and the World Wide Web Consortium (W3C). All system components are loosely connected in order to enable easy replacement if needed.
The system components consist of:
• Microsoft SQL Server (Configuration / User Administration)
• Microsoft AD / ADFS (user management)
• GeoNetwork (Handles all metadata about each dataset)
• MapCache (+ CouchBase (NoSQL DB) (Saves maps that are frequently used)
• MapServer (Rendering a map based on raw geospatial data and making a visual map from it)
• GO Publisher from Snowflake (Provides raw geospatial data with the requested properties)
Underneath these services the two databases are found:
• PostGIS geospatial database: Contains geospatial data and non-geographical registry data
• File system with geospatial data: Contains images, rasters and an elevation model
As an end user you have the opportunity to request for many standards. An online query can be sent as follows:
• WMS – Web Map Service
• WMTS – Web Map Tiling Service
• WFS – Web Feature Service
• REST API
• CSW – Catalog Service for the Web (metadata)
• WCS – Web Coverage Service
An interpreting layer called the Switchboard takes validates the query. Upon approval, it's processed further in the system, and the result is returned in the requested format (jpg, XML, GeoJSON).
Evaluation of the Datafordeler
As a subcontractor to KMD, NorthTech had influence during the agile tender process as well as during the development process of the Datafordeler. We were part of the interaction, the matching of expectations and in the dialogue with the Agency for Digitisation, and later on also in the dialogue with the operator SDFE (The Danish Agency for Data Supply and Efficiency). In this way we gained priceless experience in working with large public tenders belonging under the Danish State's K03 - standard contract for long-term IT projects based on an agile method. We will use this experience in future similar projects, in which agile cooperation is needed and in which the special expertise that NorthTech, like other, less specialized companies possess.
NorthTech is honoured to have been a participant in such a comprehensive project hat (with 8 FTEs). According to the digital magazine Version2 the budget for the entire Basic Data Programme is approximately DKK 680 million as of May 2019, and the tender for the Datafordeler was originally originally (May 2014 for 5 years) won by KMD to DKK 121 million.
The focus on open standards and the use of Open Source is in NorthTech's DNA.
Until now there are no finished solutions for a project such as the Datafordeler. NorthTech possesses the knowledge that enables us to connect a large number of standards and open source components with proprietary code, enabling a solution with a focus on the good user experience.
Together with the vision that basic data must be the public sector's common management base of high quality, which is efficiently updated in one place and can be used by everyone - also by private companies - we find it very attractive that the Datafordeler provides 24/7 accessibility, free use for all, as well as the fact that it eases the cooperation between authorities, businesses and citizens.
It has been a pleasure for us to participate in the organizing of data on a new joint platform, that provides coherent datasets and standardized based on the same principles.