The project in five facts.
Challenge, approach and results.
The diploma thesis addresses the implementation of a data pipeline in a small company KG-EKO, d. o. o. with the aim of improving processing efficiency and data reliability as well as automated report generation. The existing state is presented, in which the company uses manual data entry and processing in Microsoft Excel for daily operations, which proves to be time-consuming and limited for further analysis when dealing with larger data volumes. Within the framework of the diploma thesis, a data pipeline was designed and implemented based on the ETL (Extract, Transform, Load) pattern. The results showed that automation brings significant time savings, with PostgreSQL being more suitable for larger data volumes, whilst Pandas is better suited for smaller projects and development phases. An important part of the thesis is also dedicated to data visualisation.
Problem
The diploma thesis addresses the implementation of a data pipeline in a small company KG-EKO, d. o. o. with the aim of improving processing efficiency and data reliability as well as automated report generation. The existing state is presented, in which the company uses manual data entry and processing in Microsoft Excel for daily operations, which proves to be time-consuming and limited for further analysis when dealing with larger data volumes. Analysis of the existing state revealed that data were not adequately prepared for automated processing due to inconsistent records, missing values and inappropriate structure.
Such a state increases the probability of errors, reduces reporting reliability and limits data traceability, which represents an additional risk in business processes, particularly in invoice issuance. Manual data processing is, moreover, time-inefficient and does not enable easy introduction of automated reporting.
Solution and Method
Within the framework of the diploma thesis, a data pipeline was designed and implemented based on the ETL (Extract, Transform, Load) pattern. Data are captured through a controlled input form developed in the Python programming language, whereby validation, standardisation and logical constraints are already executed during entry. The cleaned data are then used for automated report generation, whilst simultaneously significantly reducing processing time.
A comparison of three data processing approaches was carried out: manual processing in Excel, processing with the Pandas library, and processing in the PostgreSQL relational database.
Results
The results showed that automation brings significant time savings, with PostgreSQL being more suitable for larger data volumes, whilst Pandas is better suited for smaller projects and development phases. An important part of the thesis is also dedicated to data visualisation. Dashboards were developed in Microsoft Power BI and Tableau tools, whereby Power BI proved to be a more suitable solution for a small company based on practical use.
The concluding part of the thesis addresses data pipeline security with emphasis on SQL injection vulnerabilities and confirms the importance of consistent use of established security practices. Particular emphasis is placed on proper handling of input data and the use of parameterised queries, which significantly reduce the risk of abuse, as well as the importance of consistent adherence to security guidelines in all phases of data processing.
Module, programme and stories.
Your diploma project for a real company.
A diploma thesis at Academia solves a challenge from a company – often the one where you do your 800 hours of placement. Read about the Podatkovna znanost module or book a consultation.