13.03.2025 Data Engineering Lecture

Also see github repository, created during the workshop

VS Code

A lightweight, powerful, and open-source code editor developed by Microsoft. It supports numerous programming languages natively and offers a vast ecosystem of extensions for other languages, tools, and features.

See an online tutorial for beginners:

• IDE - supports running/debugging code by clicking a button (needs a bit of a setup)
• Layout
  • terminal (on Windows can be a bit of a challenge)
  • git integration
  • file explorer
  • search (from file, from project)
• addons/extensions (dev containers, docker, remote development, remote explorer, remote - ssh)

Docker

A platform for developing, shipping, and running applications inside lightweight, portable containers. Docker simplifies the deployment of applications because it allows them to run in isolated environments.

See an online tutorial for beginners (also contains a fair bit of Linux bash tutorial as well):

• Images (docker hub) (packaged environment with (specific) software)
• Containers (running image)
• Dockerfiles (build your own images)
• Install Apache superset dockerised version
  • Apache Superset on DockerHub

Git

A distributed version control system that is essential for tracking changes in source code during software development. It supports collaboration by allowing multiple developers to work on different features simultaneously.

See an online tutorial for beginners (mostly done on the command line):

• Repositories
  • Creating, github.com, VSCode
• Branches
• Commits
  • Merging
    • Conflicts ...

Dev-Containers

Using the Remote - development extension to develop inside a Docker container allows developers to use VS Code as their interface while interacting with a consistent, containerized development environment.

See online tutorial:

• .devcontainer/devcontainer.json
• from image
• from Dockerfile
• mounts