The project in five facts.
Challenge, approach and results.
The diploma thesis presents a project in which a data flow was set up in the Azure cloud environment. The data flow began on the source database, which was Azure SQL Database. On this database, the Debezium tool was set up, which captured changes on the configured tables on this source database and forwarded these changes to a streaming tool, in this case Azure Event Hubs. This hypothesis was confirmed, as it was found that streaming tools at the three largest cloud providers have very similar functions and behaviour. Subsequently, databases in the Azure cloud environment were investigated, whereby hypothesis H2 was that Synapse Serverless SQL Pool is the most suitable for data warehousing.
Problem
The diploma thesis presents a project in which a data flow was set up in the Azure cloud environment. The data flow began on the source database, which was Azure SQL Database.
Solution and Method
On this database, the Debezium tool was set up, which captured changes on the configured tables on this source database and forwarded these changes to a streaming tool, in this case Azure Event Hubs. The arrival of a message in the Azure Event Hubs instance triggered an Azure Functions instance, which enables code execution without a server. There, a programme written in the Python programming language was triggered, which parsed the data and stored it in the Synapse Dedicated SQL Pool sink database.
In the sink database, the data was stored in several different ways of storing historical data, then SQL queries were compiled to evaluate the performance of these methods. Related to this project, four hypotheses were investigated. The first hypothesis, H1, was that data streaming tools in different cloud services are very similar.
Results
This hypothesis was confirmed, as it was found that streaming tools at the three largest cloud providers have very similar functions and behaviour. Subsequently, databases in the Azure cloud environment were investigated, whereby hypothesis H2 was that Synapse Serverless SQL Pool is the most suitable for data warehousing. This hypothesis was refuted, as it was found that Synapse Dedicated SQL Pool contains all the functionalities of Serverless SQL Pool, but in addition also enables the use of a standard database, which was mandatory in their case, and that in other cases it is also better to have a standard database, if nothing else, for administrative tables.
As mentioned in the project description, forms of storing historical data were also investigated, whereby hypothesis H3 was that SCD 6 is the most suitable. This hypothesis was refuted, as it was found that SCD 2 is equivalent in all test cases conceived, but at the same time much better for data insertion. The final hypothesis, H4, was that relational databases will maintain their dominance and that new types of databases, such as vector and NoSQL databases, will not replace relational databases.
This hypothesis was confirmed.
Module, programme and stories.
Your diploma project for a real company.
A diploma thesis at Academia solves a challenge from a company – often the one where you do your 800 hours of placement. Read about the Programsko inženirstvo module or book a consultation.