I made several crawlers and tested them in the local mongo, but I couldn’t carry out the monitoring. I wanted to hang them on ariyun, but mongodb has few remote links. some of them are either very complicated to read or very simple to write and don’t know how to operate ~
I am a novice, although I also despise the Shinshou Party, but there are some problems that I cannot help ~
Is there a more systematic answer? For example, 1,2,3,4, don’t be too complicated, I have a bad foundation ~
What does mongodb’s remote links mean? You can install mongodb on the server yourself.
Crawler mainly requires high downlink bandwidth. In fact, it is not a good choice to use external network servers. Once blocked, IP cannot be changed.
Before, I crawled through several machines in the intranet, and then mapped the ports in the intranet to view the status from outside.
Finally, the index is crawled and stored and then synchronized to the external network.