Parsing baseball files in Rust instead of Python for an 8x speedup(gregstoll.wordpress.com)
gregstoll.wordpress.com
Parsing baseball files in Rust instead of Python for an 8x speedup
https://gregstoll.wordpress.com/2021/01/10/parsing-baseball-files-in-rust-instead-of-python-for-an-8x-speedup/
7 comments
OP here. I went ahead and tried using the slicing instead of startswith() a few places, and moving the regexes out to module variables. (thanks for the suggestions!) It did speed up, but just a little bit (from ~32 secs to ~31 secs).
I may try changing the "not doneParsingEvent" checks into something that uses "continue", I'd imagine that would probably help. (although the Rust code is structured the same way) Another thing I've considered is looking at the data and figuring out which cases are more common and checking for those first...
Another thing I considered is
I may try changing the "not doneParsingEvent" checks into something that uses "continue", I'd imagine that would probably help. (although the Rust code is structured the same way) Another thing I've considered is looking at the data and figuring out which cases are more common and checking for those first...
Another thing I considered is
I read through your previous posts.
I think the double caching is why "some more cleanup to cache compiled regular expressions that I thought would also speed things up a little" didn't actually speed things up - it was already cached. (See Python's re.py in def _compile(): where it uses '_cache'.)
Did you post something where you profiled your code? I didn't see it. The cprofile module gives a decent first-pass at figuring out bottlenecks, and it's sometimes not what you think it is.
If I find that the time is spent in one or a handful of functions, and the slow-down still isn't obvious, I'll then use line_profiler for more fine-grained profiling.
There are other profiling tools, but I haven't need to use them for many years for my current projects.
I think the double caching is why "some more cleanup to cache compiled regular expressions that I thought would also speed things up a little" didn't actually speed things up - it was already cached. (See Python's re.py in def _compile(): where it uses '_cache'.)
Did you post something where you profiled your code? I didn't see it. The cprofile module gives a decent first-pass at figuring out bottlenecks, and it's sometimes not what you think it is.
If I find that the time is spent in one or a handful of functions, and the slow-down still isn't obvious, I'll then use line_profiler for more fine-grained profiling.
There are other profiling tools, but I haven't need to use them for many years for my current projects.
Yup, that makes a lot of sense - I didn't realize Python cached so many previous regex's, which is nice!
I did use cprofile a while back and I think at the time it showed a lot of time in the regexp matching. (I don't think I posted about it though) Honestly, once I implemented the parallelism I got less motivated to make it faster. But now I am kinda curious what the speed difference is between Rust and Python if I spend time trying to optimize both.
Thanks for the suggestions!
I did use cprofile a while back and I think at the time it showed a lot of time in the regexp matching. (I don't think I posted about it though) Honestly, once I implemented the parallelism I got less motivated to make it faster. But now I am kinda curious what the speed difference is between Rust and Python if I spend time trying to optimize both.
Thanks for the suggestions!
Rust should be a lot faster than Python if your time is mostly spent parsing the contents of those lines. Think that each Python op-code will be running extra assembly instructions, just to handle the virtual machine overhead.
I see a number of micro-optimizations that may give you a few percent more in Python.
For example, you use GameSituation as a mutable way to maintain parse state. You modify it with things like "gameSituation.outs += 1".
Mutating instance attributes has a much higher overhead in CPython than in C/C++ (and presumably Rust). You can reduce some of that overhead by telling the class which slots to have.
Consider "spam.py" containing the following:
Another micro-optimization is to reduce the number of temporary strings. Consider:
(BTW, there appears to be a bug in your original code, since "IW+" is 3 letters long.)
Another BTW, you might change "for line in f.readlines()" to "for line in f". Shouldn't affect performance but should reduce your overall memory use.
In closing, character-level string processing in CPython is slow so I doubt you'll get all that much faster.
You might try pypy, but with the number of temporary strings you create, my guess is pypy still won't be that much faster. Should be easy to test though.
I see a number of micro-optimizations that may give you a few percent more in Python.
For example, you use GameSituation as a mutable way to maintain parse state. You modify it with things like "gameSituation.outs += 1".
Mutating instance attributes has a much higher overhead in CPython than in C/C++ (and presumably Rust). You can reduce some of that overhead by telling the class which slots to have.
Consider "spam.py" containing the following:
class Foo:
def __init__(self):
self.a = 0
class Bar(Foo):
__slots__ = ("a",)
% python -m timeit -s 'import spam; x=spam.Foo()' 'x.a = 3'
5000000 loops, best of 5: 41 nsec per loop
% python -m timeit -s 'import spam; x=spam.Bar()' 'x.a = 3'
10000000 loops, best of 5: 32.4 nsec per loop
% python -m timeit -s 'import spam; x={}' 'x["a"] = 3'Another micro-optimization is to reduce the number of temporary strings. Consider:
if (batterEvent.startswith('W+') or batterEvent.startswith('IW+') or batterEvent.startswith('I+')):
tempEvent = batterEvent[2:]
if batterEvent[i:i+2] in ("W+", "I+") or batterEvent[i:i+3] == "IW+":
i += 2
(BTW, there appears to be a bug in your original code, since "IW+" is 3 letters long.)
Another BTW, you might change "for line in f.readlines()" to "for line in f". Shouldn't affect performance but should reduce your overall memory use.
In closing, character-level string processing in CPython is slow so I doubt you'll get all that much faster.
You might try pypy, but with the number of temporary strings you create, my guess is pypy still won't be that much faster. Should be easy to test though.
I noticed in the git repository's Cargo.toml that no optimizations are enabled.
If you add opt-level=3 to the [profile.release] section, you might be able to speed up your rust code a little more.
If you add opt-level=3 to the [profile.release] section, you might be able to speed up your rust code a little more.
Thanks! Just gave this a try and it didn't seem to help, but I'll remember for next time :-)
The "not doneParsingEvent" is used to emulate a "continue" statement. As is, every line can trigger a large number if "if (not doneParsing):" tests.
Python's s.startswith() methods are very attractive, but slow compared to the error-prone:
Here are the timings:
There are a number of uses of:
where "getRe()" implements a simple regex cache. However 1) re.match() also implements a simple regex cache, and 2) I typically implement this as a per-module variable, one per regex:
which is then used as:
This would reduce the call overhead by removing the double cache table lookup.
Bear in mind that I don't know where the time is spent (so I may be looking in the wrong place) and I don't think these changes could get an 8x performance improvement.