I have a crawler to download image from a site (https://github.com/Nandaka/PixivUtil2).
It save image as below structure:
-root - artist_id_NameA : - image_id_DescriptionX.jpg - image_id_DescriptionY.png - ... - artist_id_NameB : - image_id_DescriptionX.jpg - image_id_DescriptionY.png - ...
The problem is :
1 .When I want to batch rename artist folder name with his latest name, or I want image has some tag for searching. I have to query botn db and file system, apply rename .
Crawler restart, image url changed or somehthing else may result to some images in db but not in file system , or vice verse . Even if some db recode marked as downloaded, I still need check if that file does exist in file system.
I need images in file system for easy viewing , so can not save to db. But file system can’t store extra data such as
relationor something else .
I know all above can be solved by coding, but it indeed need much effort.
A lot of things are doing twice in db and filesystem. Such as rename:
- name artist_A change to artist_Z
- crawler get page and save new name artist_Z to db
- crawler check db, what is the artist_Z previous name, search that name in file system, rename the folder (it has many issue would occur in this step)
It would be much more convenient if db can sync with file system.
So in this situation, I wonder if there are some existing solutions to put db sync/integrete with file system ?
✓ Extra quality
ExtraProxies brings the best proxy quality for you with our private and reliable proxies
✓ Extra anonymity
Top level of anonymity and 100% safe proxies – this is what you get with every proxy package
✓ Extra speed
1,ooo mb/s proxy servers speed – we are way better than others – just enjoy our proxies!
USA proxy location
We offer premium quality USA private proxies – the most essential proxies you can ever want from USA
Our proxies have TOP level of anonymity + Elite quality, so you are always safe and secure with your proxies
Use your proxies as much as you want – we have no limits for data transfer and bandwidth, unlimited usage!
Superb fast proxy servers with 1,000 mb/s speed – sit back and enjoy your lightning fast private proxies!
99,9% servers uptime
Alive and working proxies all the time – we are taking care of our servers so you can use them without any problems
No usage restrictions
You have freedom to use your proxies with every software, browser or website you want without restrictions
Perfect for SEO
We are 100% friendly with all SEO tasks as well as internet marketing – feel the power with our proxies
Buy more proxies and get better price – we offer various proxy packages with great deals and discounts
We are working 24/7 to bring the best proxy experience for you – we are glad to help and assist you!