geolba/tethys.backend
2
1
Fork 1
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

„C15 Technical Infrastructure“ ändern

Brus 2023-05-10 13:09:51 +00:00
parent ae808a79bd
commit 487a010474
1 changed files with 31 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

37
C15-Technical-Infrastructure.md

@ -1,14 +1,39 @@
##### Technical infrastructure
R15. The repository functions on well-supported operating systems and other core infrastructural software and is using hardware and software technologies appropriate to the services it provides to its Designated Community.
The basic technical structure of TETHYS corresponds to a three-tiered client/server architecture with a number of clients and middleware components controlling the information flow and quality. On the server side a RDBMS (PostgreSQL) is used for information storage (metadata, access control lists, file checksums). All metadata is replicated into middleware/frontend systems (Solr) for fast access and search capabilities. All public interfaces on the client side (tethys.frontend) to the information system are standards compliant (W3C, ISO, OGC) and are based on web services (REST API and SOLR search services). By using modern technologies like Bulma, Typescript, Vue3, and Webpack, the TETHYS frontend is able to provide a more engaging and efficient user experience while also improving overall performance. These technologies are widely used in the industry, so it is likely that the frontend will be compatible with a wide range of browsers and devices. The TETHYS REST Service, which is known as tethys.api, is built on top of the Express framework. Express is a popular web framework for Node.js that provides a minimalist set of features for deploying RESTful APIs that are easy to maintain and scale.
The basic technical structure of TETHYS corresponds to a client/server architecture with a number of clients and middleware components controlling the information flow and quality. On the server side a RDBMS (PostgreSQL) is used for information storage. For better performance high volume and binary data (e.g. geophysics, pictures) are stored in consistent formats on hard disk arrays. All public interfaces to the information system are standards conform (W3C, ISO, OGC) and are based on web services. Metadata are dynamically marshalled from the RDB to a TETHYS specific metadata format, and transformed (XSLT, XML to JSON transform) by the frontend to various content standards (DataCite XML, Dublin Core XML, but also more community specific ones like ISO19115/ISO19139, Darwin Core). They are disseminated via OAI-PMH (https://tethys.at/oai), HTTP content negotiation (based on DOI standards), or other protocols. The Tethys ticket system used for data submissions, general user requests, or bug reports is Gitea, hostet on Geosphere.at (https://gitea.geologie.ac.at/geolba/tethys.backend/issues).
In order to comply with international metadata standards such as Dublin Core, DataCite and ISO 19139, the relevant information is delivered directly from the database by mapping (XSLT Transformation) on the fly (https://www.tethys.at/oai?verb=ListMetadataFormats). By mapping Tethys metadata to these international standards, it becomes easier for other systems to exchange and integrate metadata, which in turn can facilitate the discovery, access, and reuse of data and other resources (e.g. OGC CSW-Metadata Harvesting (ISO19139) from the Tethys RDR OAI-PMH base URL: https://www.tethys.at/oai?).
Availability is ensured by general IT monitoring to ensure connectivity to the community. The access statistics are monitored with matomo so that the server can be adjusted accordingly if necessary.
The new TETHYS editorial system on the server side, which is also known as tethys.backend, is a web-based open-source software that operates directly on the PostgreSQL database. It is built using AdonisJS, which is a Node.js-based web framework that provides a robust set of tools and features for building scalable and secure web applications. AdonisJS also includes an object-relational mapper (ORM) that allows developers to work with database tables and records using object-oriented programming techniques. The styling of the TETHYS backend is built using Tailwind, which is a utility-first CSS framework.
All basic hard- and software services are supplied by the computer center of the „GeoSphere Austria“. Most backend/middleware systems and all front end web servers / search engine are running on virtual machines (VMware), supplying sufficient capacity and performance, as well as high availability due to virtualization. The Postgres backend database is running on a dedicated machine supplied with a high performance IO system. Machines are operated with Linux systems (Ubuntu). The editorial system is running as a client/server system on a Ubuntu 22.04 LTS (VMware).
Hardware is generally renewed every 3-4 years, which is transparent to the system because of virtualization. Operating systems are regularly updated to the latest releases and patches. The central archival storage system replicates up to 65 TB of data into two robotic tape archives ?(TYP)?, which are located in separate buildings. The backup facilities work on high capacity Linear Tape Open LTO-tape drives and media installed in this ?TYP? environment. The IT has support contracts for all relevant hard- and non-open-source software products with ?5 days business hours response, part replacement 1 day?. Proper functioning of Tethys related systems, applications, services, and processes is also ensured by usage of a professional monitoring software like ICINGA for IT infrastructure.
General descriptions of the systems and software used can be found in our Wiki. There you will find public information about the recovery of the Tethys research repository, details about the database model, instructions for starting docker container and a data architecture diagram for a clear understanding of all storage locations. All code repositories are accessible online via a Gitea instance hosted at the GeoSphere Austria data center. The associated Tethys Docker images are also securely stored there using the built-in Docker registry functionality. All necessary configurations to launch the Docker container are described in the wiki.
Internal information about virtual server, maintenance and security settings are stored in a separate, private wiki on internal LAN servers.
Hardware Infrastructure is generally renewed every 3-4 years, which is transparent to the system because of virtualization. Operating systems are regularly updated to the latest releases and patches.
General descriptions of used systems and software are supplied within our Wiki ?(GiteaWiki, https://gitea.geologie.ac.at/geolba/tethys.backend/wiki/?_Page)?, which is currently under revision. A second Wiki (https://gitea.geologie.ac.at/geolba/tethys.intern/wiki) is operated separately from the other Gitea infrastructure for disaster safety and recovery containing internal information about servers, installation, maintenance and relationships. The technical Wiki also covers instructions for running own services, databases, and topics like VM snapshots and backup.
GeoSphere Austria is certified according to the international standard ISO 27001 for information security management. Compliance with these standards is monitored by the organization responsible for issuing certification and conducting audits. This organization inspects and monitors the organization's infrastructure and processes to ensure that the relevant standards are met and the certification for GeoSphere Austria can be renewed and compliance is monitored.
All the software components used for the three-tiered client/server architecture in the TEHTYS research data repository are versioned through GIT and can be accessed online via a Gitea instance hosted by the computer center of GeoSphere Austria. The frontend can be accessed at https://gitea.geologie.ac.at/geolba/tethys.frontend, the backend at https://gitea.geologie.ac.at/geolba/tethys.backend, and the API at https://gitea.geologie.ac.at/geolba/tethys.api.
The TETHYS research repository is being developed using the Continuous Integration/Continuous Deployment (CI/CD) practice, which involves frequent testing and deployment of code changes to production. Whenever new code is published, automatic tests run in the background (CI), and when new versions are released, new Docker images are automatically deployed (CD). This entire process is implemented using Gitea Actions, a powerful platform that enables automation of various tasks like building, testing, and deploying code, as well as sending notifications such as alerts for failed tests. All workflows are defined in YAML files, such as the ci.yaml file, which is always triggered whenever new code is committed to the TETHYS backend.
To ensure that the availability, bandwidth, and connectivity are sufficient to meet the needs of the designated community of TETHYS RDR, the following measures are taken:
* A reliable and reputable hosting provider: All basic hard- and software services of TETHYS repository are hosted by the Geosphere Austria computer center, which is a reliable provider that supplies the necessary infrastructure and software services.
* All the necessary backend and middleware systems for TEHTYS, as well as the web servers and SOLR search server for the frontend, are operating on virtual VMware servers. This setup provides sufficient capacity and high performance, as well as enhanced availability through virtualization. The dedicated machine responsible for the PostgreSQL database is equipped with a powerful IO system to ensure high performance. The server operating systems responsible for the TETHYS frontend and backend are operating on virtualized Ubuntu 22.04.
* The use of professional monitoring software like ICINGA helps to ensure that Tethys research repository is running smoothly and that its users can access the data they need without interruption. Tethys research repository utilizes the open-source web analytics software, Matomo, to monitor access statistics and gain insights into user behavior.
*
* Caching mechanisms: TETHYS REST API has implemented caching mechanisms, which are based on REDIS cache. REDIS provides fast in-memory data storage and retrieval to reduce the load on the repository's servers and speed up access to frequently accessed content.
*
* Load balancing: TETHYS uses NGINX as a load balancer to distribute traffic across multiple servers to ensure that the repository can handle high levels of traffic and provide a seamless user experience.
*
* Performance monitoring and capacity planning: TETHYS staff regularly monitors the performance of the repository and performs disk capacity planning to ensure that the repository can handle future growth in usage.
*
* Multiple access points: TETHYS provides multiple access points to the repository, including web interfaces, REST APIs, and web applications. TETHYS frontend web application employs a full responsive frontend design to ensure that users can access the data from a variety of devices and platforms. This means that the repository's web interface is optimized to provide an optimal viewing and interaction experience across a wide range of screen sizes and device types, including desktops, laptops, tablets, and smartphones.
There are several processes that are used to monitor and manage the need of technical changes in software development of Tethys.
* Change management: This involves establishing a formal process for requesting, reviewing, approving, and implementing changes to Tethys software development. Changes may include new features, bug fixes, or modifications to existing functionality.
Version control: This enables the developer team to keep track of changes to software code and ensure that everyone is working on the most up-to-date version. This is done using Git versionskontrolle.
* Continuous integration/continuous delivery (CI/CD): These are practices that involve automating the building, testing, and deployment der software. This helps catch errors and ensure that changes are released into production environments as quickly and reliably as possible.
* Agile methodologies: Agile development focuses on iterative development cycles, with frequent feedback and collaboration between developers, stakeholders, and end-users. This approach helps ensure that software development stays aligned with changing requirements and priorities.