I’m maintaining an already built system. The current system operates as following:
- Client (UI) submits a request.
- Web layer round-robins to one of backends.
- Backend takes request and handles.
- Backend can spend millis / seconds / minutes on request depending on which job it was asked to do.
so let’s say
- train-model global cluster limit – 4 (large task) limitation by shared resource like db.
- generate-model global cluster limit – 8 (large task) limitation by shared resource like db.
I want to do the least amount of work and make this system more resilient, meaning it would not loose and fail jobs and timeout.
- Every job is first placed inside a queue, for multitanency a queue per each customer.
- Backends take jobs from queue according to their capacity.
- Each time a backend takes a job it takes it from the “next” customer queue to be fare among them.
- If cluster already handles 4 train-model (cluster wide limit) I cannot take more long running tasks.
- This means that even if I have split my queue into long running tasks and short running tasks (2 queues for each customer instead of one) this would mean I would not consume also tasks from the long running queue tasks because one of the type of long running tasks there reached the global cluster capacity (like db is fully utilized shared resource).
Now if I then decide to take the train and have a queue for it because of it’s hard global cluster limitation I would have
- train-queue (heavy task)
- generate-queue (heavy task)
- short-queue (lightweight task).
So if I have hit the top ceiling of 4 trains per cluster I would take pull out tasks from the other 2 queues, the problem is that if i have now many tasks in the generate-queue and short-queue queues i then will starve the train-queue, wouldn’t i?
Is there any well known pattern to deal with it?
*Note: I cannot change the whole architecture like to introduce yarn/spark, the system is already built and coded in a way I can only change how I route the requests. When I rewrite it, I will, but now no-rewrite. All jobs are in-process, meaning, it’s not spawning new processes. (simple java nio and some multithreading).
✓ Extra quality
ExtraProxies brings the best proxy quality for you with our private and reliable proxies
✓ Extra anonymity
Top level of anonymity and 100% safe proxies – this is what you get with every proxy package
✓ Extra speed
1,ooo mb/s proxy servers speed – we are way better than others – just enjoy our proxies!
USA proxy location
We offer premium quality USA private proxies – the most essential proxies you can ever want from USA
Our proxies have TOP level of anonymity + Elite quality, so you are always safe and secure with your proxies
Use your proxies as much as you want – we have no limits for data transfer and bandwidth, unlimited usage!
Superb fast proxy servers with 1,000 mb/s speed – sit back and enjoy your lightning fast private proxies!
99,9% servers uptime
Alive and working proxies all the time – we are taking care of our servers so you can use them without any problems
No usage restrictions
You have freedom to use your proxies with every software, browser or website you want without restrictions
Perfect for SEO
We are 100% friendly with all SEO tasks as well as internet marketing – feel the power with our proxies
Buy more proxies and get better price – we offer various proxy packages with great deals and discounts
We are working 24/7 to bring the best proxy experience for you – we are glad to help and assist you!