> For the complete documentation index, see [llms.txt](https://arditxhaferi2.gitbook.io/python-web-scrapper/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://arditxhaferi2.gitbook.io/python-web-scrapper/project/news-scrapper.md).

# News Scrapper

We will continue to create a news scrapper with the knowledge we learn from our tools.

## Getting The Links

We use the homepage of the news site to get all the links of the most recent articles, so we need to analyze to find what all the links have in common this time it was the `right-post-category` class and append them all to a array. In this example we will use GazetaExpress.

```python
r = requests.get(https://www.gazetaexpress.com/zgjedhjet2021/)
soup = BeautifulSoup(r.content, 'html.parser')
data = soup.findAll(element, attrs={'class': 'right-post-category'})
for div in data:
    links = div.findAll('a')
    for link in links:
        if link.get('href'):
            linksArray.append(link.get('href'))
```

## Getting The Content

Now that we have the links of the articles we just have to find how is the content contained with which class, this case we have to get the content with the `single__content` class.

```python
for link in linksArray:
    r = requests.get(link)
    soup = BeautifulSoup(r.text, 'html.parser')
    content = soup.find('div', attrs={'class': 'single__content'})
    text = content.text.lower()
    print(text.count(keyword))
```

We can continue to use now all the articles post that we get back for research or for anything in particular. We can use the `count` function to get how many times a word was used in the articles and see a pattern or create some statistics about anything really.