Python from absolute zero. Working with OS, learning regular expressions and functions

Working with files

Let’s start, as always, with simple things. Python has a module with the laconic name os, which (you won’t believe it!) is designed for interaction between a program and the operating system, including file management.

The first thing we need to do, of course, is import it at the beginning of our script:

And now, various interesting opportunities are opening up for us. For example, we can get the path to the current folder. At first, it matches the one you were in, when you started the script (even if the script itself is located somewhere in another place), but during the program execution, we can change this value using the os.chdir() function.

Let’s try to get a list of files with the .py extension located in the current directory. For this, we use the os and fnmatch modules.

The fnmatch module allows to search for specific text in strings that matches a given template using a mask:

• * replaces any number of any characters;
• ? replaces any one character;
• [seq] replaces any characters from the sequence in square brackets;
• [!seq] replaces any characters except those in square brackets.

Let’s ruthlessly delete some file:

Let’s rename the file:

Now let’s create a folder at the specified path and immediately delete it. For this, the shutil module is useful, which has the rmtree() function, which deletes a folder along with its contents.

Let’s say you want to get a list of all files contained in folders at a given path (including subfolders too) to find something interesting. The script will look like this:

The walk() function of the os module takes one required argument — the directory name. It sequentially goes through all nested directories and returns a generator object, from which it gets:

• address of the next directory as a string;
• a list of names of first-level subdirectories for a given directory;
• list of file names in this directory.

Now I’ll show you how to find out the size of any file, as well as its modification date.

Let’s play a joke on the user: let’s create some file and constantly open it with the program that usually opens this file in the system:

Below, there is a list of some other useful commands:

• os.path.basename('path') – returns the name of the file or folder at the end of the path;
• os.path.dirname('path') – returns the parent path of the path object;
• os.path.splitext('path') – splits the path into the path and file extension;
• os.path.exists('path') – whether there is a path to the file or folder;
• os.path.isfile('path') — whether the path object is a file (existing);
• os.path.isdir('path') – whether the path object is a folder (existing).

Regular expressions

Regular expressions are special patterns to find and replace strings in text. Generally speaking, they can be considered an independent language, and its study goes beyond the scope of this loop. We’ll go over the very basics and use of regular expressions in Python.

The re module is responsible for working with regular expressions in Python. First of all, let’s import it.

As a simple pattern, we can use some word. Let it be “beer” according to tradition:

The re.search(pattern,string) command searches the text string for the first occurrence of the pattern pattern and returns a group of strings that can be accessed via the .group() method. But the search command only searches for the first occurrence of the pattern. Therefore, in our case, only one result will be returned — the word “beer”, despite the fact that it appears twice in our text.

To return all occurrences of a pattern in the text, use the re.findall(pattern, string) command. This command will return a list of strings that are present in the text and match the pattern.

In the previous two program examples, you simply used a word as the pattern to search for strings. But that’s not where the power of regular expressions lies. You can replace parts of the template with special characters so that the template matches not only specific words, but also a wide variety of strings.

Let’s, for example, try to find all the words in the text that begin with “pi”. For this we use a special symbol b – it means ’beginning of a word’. Immediately after it, we indicate what the word should begin with, and write a special symbol w, which means that some letters should follow in the template (the plus means that there can be one or more of them) until a non-letter symbol is encountered (for example, a space or punctuation mark). The template will look like this: r"\bпи\w+".

Let’s try to complete a slightly more difficult task. We will find in the text all emails with the mail.ru domain, if they are there.

As you can see, we used the same trick as last time — we wrote the special character \b to indicate the beginning of a word, then \w+, which means “one or more letters”, and then @mail.ru, escaping the period, since otherwise it would mean “any character”.

Often you need to find some element of a string surrounded by two other elements. For example, it could be a URL. To select the part of the template that needs to be returned, brackets are used. I’ll give you an example where you’ll get all the link addresses from a piece of HTML code.

The code above used the pattern r'href="(.+?)" – in this pattern, the search string starts with href=" and ends with another double quote. The parentheses are used to indicate which part of the string that matches the pattern you want to get into ерthe result variable. The period and plus inside the brackets indicate that any characters (except the newline character) can be inside the quotes. The question mark means that you should stop before the first quotation mark you encounter.

We can not only search for strings, but also replace them with something else. For example, let’s try to remove all tags from the HTML code. To do this, use the re.sub(pattern,'what to replace with',string) command.

The program will print the string without tags, since we replaced them with an empty string.

Regular expressions are very powerful things. Once you master them, you can do almost anything with strings, and when combined with Python code, literally anything. To begin with, you can experiment and change some of the recipes given.

Functions

It’s time to talk about functions in more detail. We’ve already called various functions many times, both built into Python (e.g. print()) and from plugins (e.g. urllib.request()). But what is a function from the inside and how to make them yourself?

Imagine you have some set of commands that need to be executed several times, changing only the input data. Such blocks of commands are usually placed in separate pieces of the program.

A function can have input parameters — these are one or more variables that are written in brackets after the function name. When you call a function, you can pass it arguments for these parameters. Some of the parameters may be optional or have a default value in case one is not passed.

A function declaration starts with the keyword def, followed by the function name, parameters in parentheses, and the program code separated by four spaces. A function can return one or more values using the return keyword. It stops the function, by the way, and if there are any commands following it, they will be skipped.

As an example, let’s look at the simplest function that will take any two numbers as arguments and multiply them, returning the result of the multiplication. Let’s call it umn.

Now that you have described a function, you can call it further in the same program.

Sometimes, you need to make one of the parameters optional by setting a default value for it.

Now, if you call a function and don’t pass it a second argument, it will simply consider it to be ten, meaning it will multiply any number passed by ten.

Even though the b parameter in this case is 10 by default and is not required to be passed as the second argument, you can still pass a second argument if you want, and then the passed value will be used as b, not 10.

Inside the program, we can call the function we created as many times as we want.

Let’s create a program that will calculate a salary increase for every vulnerability a hacker finds at work. Each hacker will have their own salary, depending on their rank, but the bonus calculation for everyone works on the principle of “+2% to the base salary for a vulnerability, if more than three such vulnerabilities are found.”

Let’s make a function that takes as arguments the employee’s salary and the number of vulnerabilities found. To round the result, we use round() function, which will round the increase to an integer.

If a function must return more than one value, you can list them separated by commas.

The function will return a list, but we can immediately assign the returned values to some variables:

Within functions, it is quite possible to use variables that were encountered in the program code before the function was called. But if you set a variable with the same name inside the function code, then this variable will automatically become local and all further changes will occur with it only within the function.

Let me explain with an example:

As a result of executing the program, you will see one. Why? Inside the function code, we assigned the variable z the value 15, and it became local, and all changes to it will occur inside the function, while in the main program, its value will still be equal to one.

It’s a little hard to understand, but it’s actually pretty handy. If you write several similar functions, you can use the same local variable names inside them without worrying that they will somehow affect each other.

Variables declared outside functions are called global. If you want to change one of them from inside the function, then declare it inside the function using the global keyword.

This program will print 8. Note that we didn’t return anything, we just changed the global variable. By the way, a function that doesn’t return anything will return the value None.

The word None will be displayed on the screen. This can be useful if the function returns something only if some conditions are met, and if they are not met, the execution does not reach return. Then you can check if it returned None.

This function checks if the value is equal to one or two, and if not, it returns None. This can be further checked using if:

So, we have learned how to create functions, call them and return parameters from them, and also use global variables inside functions. From this point on, we can already take on relatively complex examples!

Practice: Checking SQL Vulnerabilities

This time we will create a script that will search for SQL vulnerabilities on different URLs. Let’s create a urls.txt file in advance, each string of which will contain website addresses containing GET parameters. For example:

Let’s write a script that gets a list of similar URLs from our file and adds a quote to each of the GET parameters, trying to trigger SQL database errors.

In the goods.txt file, you will get a list of vulnerable sites (or nothing if none are found).

Where can I get a list of URLs to check? Google dorks are often used to find such URLs.