Recently I’ve been doing some experimenting with RPN and the shunting-yard algorithm, in order to test these systems more appropriately I planned on writing a tokenizer and then using these tokens to check validity and eventually get some output. I also think that I could use this to work with some primitive programming language, such as making a CHIP-8 assembler.
The intention is for my tokenizer to separate the input string into a list of the following:
- Individual Symbols (
'(', ')', '*', etc...)
- Sequences of digits (
'1', '384', etc...)
- Sequences of characters (
'log', 'sin', 'x', etc...)
Note that because of this sequences such as:
'3', '.', '14')
'6', '.', '02', 'E', '23')
Will not come out as the numbers they represent but can be reconstructed later on.
But sequences such as
'3x' will come out as
'3', 'x' making it easier to account for multiplication of variables.
For the most part I’m quite happy with this code, a couple things that I’m interested in (alongside general review) are:
- How can I make the line
if l.isalpha() and buf.isdigit() or l.isdigit() and buf.isalpha():more concise?
- What about the
if buf: out += [buf]; buf = ''lines? Would there be anything wrong with putting this inside a nested function in
tokenize? Or would
out, buf = out + [buf], ''be more pythonic?
- This technique makes it easier later on to identify function calls such as min, max or sin, but how would I differentiate the meanings of
x*yvs a variable actually called
xy, also this question is less relevant in the context of programming languages which would parse
'xy'as a single token over the multiplication of 2)(This question is possibly out of scope for CR, if so this question can be removed)
The reasons for these questions specifically is that I like concise code, writing it on few lines without having any too long.
def tokenize(s): out =  buf = '' for l in s: if not l.isalnum(): if buf: out += [buf] buf = '' out += [l] else: if l.isalpha() and buf.isdigit() or l.isdigit() and buf.isalpha(): out += [buf] buf = '' buf += l if buf: out += [buf] return out
✓ Extra quality
ExtraProxies brings the best proxy quality for you with our private and reliable proxies
✓ Extra anonymity
Top level of anonymity and 100% safe proxies – this is what you get with every proxy package
✓ Extra speed
1,ooo mb/s proxy servers speed – we are way better than others – just enjoy our proxies!
USA proxy location
We offer premium quality USA private proxies – the most essential proxies you can ever want from USA
Our proxies have TOP level of anonymity + Elite quality, so you are always safe and secure with your proxies
Use your proxies as much as you want – we have no limits for data transfer and bandwidth, unlimited usage!
Superb fast proxy servers with 1,000 mb/s speed – sit back and enjoy your lightning fast private proxies!
99,9% servers uptime
Alive and working proxies all the time – we are taking care of our servers so you can use them without any problems
No usage restrictions
You have freedom to use your proxies with every software, browser or website you want without restrictions
Perfect for SEO
We are 100% friendly with all SEO tasks as well as internet marketing – feel the power with our proxies
Buy more proxies and get better price – we offer various proxy packages with great deals and discounts
We are working 24/7 to bring the best proxy experience for you – we are glad to help and assist you!