News crawler website
- Platform: Any - browser based
- Language: Javascript, PHP
- Other used: Simple HTML DOM parser, Youtube data API, Wordpress
Idea
The main idea was to create one simple website that lists all of the hungarian news about cryptocurrencies from multiple sites. Meanwhile I discovered, there are much more news sources than just written media, so some youtube channels were also included in the list.
How it works
The background scripts automatically parse the main news portals in every hour, seeking for new content. If any fresh article/video found on the observed sites or on the Youtube channels, then they will be stored in the database.
Purpose of the project
The main database is growing day by day continuously, in the beginning of 2021 the number of stored articles+videos exceeded the number 20,000. Visitors come regularly for fresh content, and the feedbacks are very positive. This answers the project's main question: "will anyone be interested in hungarian crypto news?"
After two years, all of the above led to a complete rewrite of the code from scratch, and added Wordpress as a frontend in 2020. Fresh articles/videos are stored as WP posts, with the help of some background PHP scripts and a little bit of magic :)
In 2023 february, the number of stored news is over 35,000...
This was version 3.0 which was bought with the webpage by another crypto related media portal in 2021.
Since then, the v4.0 engine is under (not so active) testing at https://onecrypto.news/ and parsing english crypto news continuously without any control or supervision. I has a more modular and customizable structure than the previous ones. Instead of cronjobs, a continuously running background process is controlling the parsing threads, with the help of a self monitoring system to prevent stops. The parsing process and the website is running automatically without interruptions since 2021.
Also every news are posted instantly to the relevant FB page too: https://www.facebook.com/www.onecrypto.news
Twitter posting was also implemented, but the twitter AI always thinks the frequent posts are just spam and blocks the account regularly.
Posting to discord is implemented, and working without issues. (DC server is not public at the moment)
Links
Webpage: kriptokereso.com
History
A screenshot from the past:
This was the first version at day 1. No Wordpress, no CMS, just a bunch of PHP scripts to parse data from a few websites. :D And a downloaded free HTML template.