Garanti del Talento

dgt_bash03 – File and data manipulation

Module Title: dgt_bash03 – File and data manipulation

Description:

The module “dgt_bash03 – File and data manipulation” is an advanced module on file and data management with Bash. It is intended for system administrators, developers, and IT professionals who work with command-line data processing and want to use it to automate complex tasks through natural language.

Module Overview:

In this module, participants study text processing tools such as sed (stream editor) and awk, which are essential for manipulating data in Unix-like environments. You’ll learn how these tools operate on files and turn raw data into structured information. In practice, you will describe the transformation you need to the machine (“from this access log, count requests per status code and sort them”), then read the one-liner or script it writes, predict its output from what you know about patterns and fields, and verify it on real data before trusting it.

Key Learning Objectives:

  1. File Management:
  2. Understand the basics of file creation, including how empty files are created with commands like touch and redirection operators.

  3. Explore advanced techniques for file manipulation and organization, and learn to ask the machine to apply them safely.

  4. Data Manipulation with sed:

  5. Understand how sed searches, replaces, and edits text within files, so that you can read and check the commands the machine proposes.

  6. See practical examples of how sed scripts automate data transformation tasks, and learn to spot an expression that would change more than intended.

  7. Data Visualization with AWK:

  8. Learn the main features of awk, a programming language designed for pattern scanning and processing.

  9. Learn how awk extracts, formats, and analyzes data from text files, so that you can direct the machine to generate reports and summaries and validate the numbers it returns.

  10. Practical Applications:

  11. Engage with real-world scenarios where these tools can be applied to solve complex problems.

  12. Work on natural-language exercises: instruct the machine, review the commands it writes, and verify the results against the theory.

  13. Advanced Techniques:

  14. Explore advanced sed and awk functionalities, including regular expressions and scripting capabilities.
  15. Understand how these tools fit into larger Bash scripts for complete data processing solutions.

Target Audience:

This module is intended for anyone working with file manipulation and data processing on Unix-based systems. It covers the fundamentals for beginners as well as topics relevant to experienced professionals.

Learning Outcomes:

By the end of this module, participants will understand the essential techniques of Bash file and data management and will be able to have the machine handle large datasets and automate repetitive tasks, while checking that every transformation is correct.

The module provides a working knowledge of sed and AWK and of how command-line tools are used to manage data in a professional environment.
The students can push their exercises to the Academy DevOps & SRE Git project. For this module, create a folder with your username as its name in the following subfolder: https://github.com/Garanti-Del-Talento/gdt_academy