Python from absolute zero. Learning to work with strings, files, and the Internet

Let’s start with the strings. To solve the problem that the friends faced, Cheburashka used the replace() function, which replaces one substring in a string with another.

First, he declared a variable s and placed there the string that Gena sent him.

Next, Cheburashka determined a dictionary of words that needed to be replaced.

And now, using the for loop, Cheburashka went through the dictionary to replace each word (key) with the corresponding value from the dictionary (slova[key]):

The replace() function is useful for completely removing some words from a string. To do this, we will replace them with an empty string (if you open and close the quotation mark, you will get an empty string):

To get the number of characters in a string, use the len() function.

And, as I already said in the previous article, you can take slices from strings as from arrays if you specify the beginning and end of the substring in square brackets after the variable. The position starts from zero.

If you need to make a cut from the beginning of the string, you don’t need to write the first digit.

Let’s say you need to find strings in a list that start with https. We iterate over them using for, for each, we check whether the first five characters match the string https, and if so, we output the string:

To count the number of occurrences of a substring in a string, you can use the .count() method:

Sometimes, there may be extra spaces or string breaks at the beginning or end of a string. Let’s remove them with a special command .strip():

To determine whether a substring exists in a string s, you can use the .find() method:

If the desired substring is found, then its position in the string will be stored in the variable n , and if not found, n will be equal to -1.

Let’s try to determine if the string contains an email address from Xakep.ru, that is, we will look for the substring @xakep.ru.

But first, we need one more string method – .split(). It allows to split a string into parts by specifying a separator string as an argument. For example, s.split('\n') will split the text into paragraphs based on the string break character. If you leave the brackets empty, the default separator, a space, will be used.

The .join() method, on the contrary, allows to glue strings together. It takes a list and returns a string where each element of the list is connected to the other through the string you called this method on.

Formatting strings

We have printed various things many times by connecting strings with simple addition. This is not always convenient, especially considering that if you come across numbers, you will have to convert them into strings using the str() function. There is a more beautiful and convenient way to substitute variable values into strings. More precisely, two slightly different methods.

Method 1 – using the .format() method

We can insert a pair of curly brackets into the string, and then call the string’s .format() method and pass it the desired values in the order they are substituted into the string.

You can pass information as a list separated by an asterisk:

Method 2 – via f-strings

Another option is to write the letter f before the string and then specify the variables directly in curly brackets.

The main advantage of this method is that you can insert a value into a string multiple times. In addition, you can change the values directly in the curly brackets: Python will first perform all the actions in them, and then substitute the resulting value into the string. So, the .upper() method in the example above makes all letters uppercase.

Files

The methods listed are enough to allow you to do whatever you want with strings. But where will these strings come from? Most often they are written in files, so now I will tell you how to manage them in Python.

To work with a file, you need to open it. The open() function is used for this, and it works like this:

There are several modes of working with files, but you are mainly interested in:

• r — open a file to read information from it;
• w — open a file to write information to it (creates a new file);
• a — open a file to append information to the end of the file (appends information to the end of an existing file);
• a+ — additional writing and reading.

To avoid problems with paths in Windows, use double slashes \\ in them, and also put the letter u before the opening quote of the file path, indicating that the string is in Unicode encoding:

You can read string from a file using the .read() method:

Alternatively, you can sequentially read individual string from the file using a for loop:

Once you have finished working with the file, you need to close it.

Let’s now try to create a new text file in the same directory as our script and write the values of some variables into it.

Please note that at the end of each string there is a \n symbol — a transition to a new string.

Let’s say you want to add a third string to the end of this file. This is where the re-recording mode comes in handy!

To open files, it is also very convenient to use the with open('file name with path and extension', 'file mode') as f construction, because thanks to the with word, the file will be closed automatically and you won’t have to think about it.

Working with the web

Let’s learn how to get information from web pages. First, you need to install several modules. We write in the command line:

The requests module allows to make GET and POST requests to web pages. The html2text module is used to convert HTML code of web pages into plain text, that is, it cleans it from HTML tags.

We import our new modules at the beginning of the program and try to get some page from the Internet.

The program will print out a lot of HTML code that makes up the magazine’s main page. But what if you just want the site text, not a jumble of tags? Here html2text will help. It will extract text, headings and images from the code and return them without HTML tags.

In addition to GET requests, there are so-called POST requests, which are used to send large texts or files to the server. If you see a form on a website, especially with a file upload, then most likely, when you click the “Submit” button, a POST request will be made.

The requests library also allows to make POST requests. You may find this useful for simulating user actions — for example, if you need to automate work with a website. You can even use this as a home-made Burp alternative!

Let’s see how to send a regular POST request. Let’s assume that there is a guest.php script on the site.ru website, which accepts the user name name and message message from the form via a POST request, and then posts them to the guestbook.

Now let’s send a request with the payload.php file as an attachment and the same two form fields as in the previous request. The file will come to the server under the name misc.php.

All that’s left is to learn how to download files. This is a lot like requesting pages, but it’s best to do it in streaming mode (stream=True). We will also need the shutil module, which has a convenient copyfileobj function. It allows to copy the contents of binary files — in our case, from the Internet to our disk.

Error handling

Before I look at a more real-world example, I need to show you one more language construct that is indispensable when working with files and the network. This is handling exceptional situations, that is, errors.

Often, when running a program, the computer encounters various problems. For example, file not found, network unavailable, disk space out. If the programmer has not taken care of this, the Python interpreter will simply exit with an error. But there is a way to anticipate problems right in the code and continue working — the try... except construct.

It looks like this:

You can catch specific types of errors by specifying the type name after the except keyword. For example, KeyboardInterrupt is triggered if the user tries to terminate a program by pressing Ctrl-C. It is in our power to prohibit this from happening!

Heck, we can even allow division by zero if we catch the ZeroDivisionError error. This is what it will look like:

Writing a port scanner

Now we’ll write our own port scanner! It will be simple, but quite functional. The socket module, which implements work with sockets, will help us with this.

This is what the code will look like.

As you can see, nothing complicated!

Homework

1. Make the port scanner get the list of IPs from one file and write the scan results to another.

2. In the previous article, you learned how to work with the clipboard. Write a program that continuously runs and periodically receives the clipboard contents. If it has changed, it adds it to the end of the monitoring.txt. Try to log only those intercepted strings that contain Latin letters and numbers, this way you are more likely to catch passwords.

3. Write a program that reads a file of this type:

   Ivan Ivanov|ivanov@mail.ru|Password123

   Dima Lapushok|superman1993@xakep.ru|1993superman

   Vasya Pupkin|pupok@yandex.ru|qwerty12345

   Frodo Baggins|Frodo@mail.ru|MoRdOr100500

   Kevin Mitnick|kevin@xakep.ru|dontcrackitplease

   User Userson|uswer@yandex.ru|aaaa321

   The program should sort the strings by domains from the email, create a file for each domain, and place a list of email addresses in each file.

4. Write a program that goes through the sites on the list, downloads the robots.txt and sitemap.xml files and saves them to disk. If the file is not found, a message about this is displayed.

That’s all for today. In the next article, you’ll learn how to work with the OS file system, understand functions, discover the power of regular expressions, and write a simple SQL vulnerability scanner. Don’t miss it!