Python:Speed - Revision history

KevinYager: /* Make it faster */

2019-09-08T16:19:44Z

‎Make it faster

KevinYager: /* Make it faster */

2019-07-25T12:48:29Z

‎Make it faster

KevinYager: /* Make it faster */

2019-07-25T12:45:31Z

‎Make it faster

KevinYager at 12:39, 25 July 2019

2019-07-25T12:39:00Z

KevinYager at 12:44, 21 June 2019

2019-06-21T12:44:15Z

KevinYager: /* Make it faster */

2019-06-21T12:43:24Z

‎Make it faster

KevinYager: /* Make it faster */

2019-06-21T12:39:57Z

‎Make it faster

KevinYager at 12:38, 21 June 2019

2019-06-21T12:38:16Z

KevinYager at 12:33, 21 June 2019

2019-06-21T12:33:11Z

Nsls2user at 20:20, 6 January 2017

2017-01-06T20:20:59Z

@@ Line 22: / Line 22: @@
 #** [http://www.celeryproject.org/ Celery] distributed task queue
 #** [https://github.com/AustralianSynchrotron/lightflow lightflow] distributed workflow
 # '''Externals''': Critical code can be written in C/C++, and called as a function within Python. This allows the computational bottleneck to be written in a more specialized and efficient manner.
 #* SWIG ([http://www.swig.org/ official site], [https://en.wikipedia.org/wiki/SWIG Wikipedia]) can provide a 50-200&times; speedup.

@@ Line 8: / Line 8: @@
 # '''Re-code''': If code is running slowly, you should identify the bottleneck in the code, and rework it. Typically, using the most appropriate algorithm can improve execution by orders-of-magnitude.
 # '''Don't worry about it''': Before spending serious effort optimizing code for speed, you should decide if it's even necessary. Does it matter if your code takes a minute or an hour to run? Oftentimes it simply isn't worth optimizing code.
 # '''Parallel''': Python has several mechanisms for adding simple parallelism to your code. For instance if you're processing hundreds of images in sequence, and have a computer with 16 cores, there's no reason you can't load and process 10 images at a time in parallel.
 #* joblib ([https://joblib.readthedocs.io/en/latest/ official site]) allows simple distribution of tasks.
@@ Line 19: / Line 24: @@
 # '''Externals''': Critical code can be written in C/C++, and called as a function within Python. This allows the computational bottleneck to be written in a more specialized and efficient manner.
 #* SWIG ([http://www.swig.org/ official site], [https://en.wikipedia.org/wiki/SWIG Wikipedia]) can provide a 50-200&times; speedup.
-#* Cython ([http://cython.org/ official site, [https://en.wikipedia.org/wiki/Cython Wikipedia]) is a version of Python with an interface for invoking C/C++ routines.
+#* '''Cython''' ([http://cython.org/ official site, [https://en.wikipedia.org/wiki/Cython Wikipedia]) is a version of Python with an interface for invoking C/C++ routines.
 #* ctypes ([https://docs.python.org/2/library/ctypes.html documentation]) is a function library that provides C-compatible data types, allowing external libraries to be used in Python.
 #* Python/C API ([http://dan.iel.fm/posts/python-c-extensions/ documentation], [http://dan.iel.fm/posts/python-c-extensions/ tutorial]) is available in Python, allowing C extensions to be directly called in Python without much overhead. This 'manual' method lacks the clean wrapping provided by the previously-enumerated methods, but is the most direct method and works well for calling small bits of code.
 # '''Translation''': There are some attempts to automatically translate Python code into optimized lower-level code.
 #* shedskin ([http://code.google.com/p/shedskin/ official site 1], [http://shedskin.github.io/ official site 2], [https://en.wikipedia.org/wiki/Shed_Skin Wikipedia]) translates Python into C++, providing a 2-200&times; speedup. Most extensions/libraries are not currently supported. On the other hand, one can isolate some critical code and convert this to an optimized external that is called from conventional Python code.

@@ Line 5: / Line 5: @@
 # '''Make it Pythonic''': Often code runs slowly simply because you are not taking full advantage of Python idioms. Raymond Hettinger has nice notes about "being Pythonic" (e.g. [https://gist.github.com/0x4D31/f0b633548d8e0cfb66ee3bea6a0deff9 notes] or [https://www.youtube.com/watch?feature=player_embedded&v=OSGv2VnC0go video]).
-# '''Libraries''': Exploiting Python libraries (which are highly optimized and often written in lower-level languages) can greatly improve performance. In particular, using numpy for matrix-style numerical computations (rather than using expensive for-loops or other iterations) can massively speedup computations. Processing on images can be improved using the Python Image Library (PIL), etc.
+# '''Libraries''': Exploiting Python libraries (which are highly optimized and often written in lower-level languages) can greatly improve performance. In particular, using [https://numpy.org/ numpy] for matrix-style numerical computations (rather than using expensive for-loops or other iterations) can massively speedup computations. [https://www.scipy.org/ Scipy] can be used for optimizations, image processing can be improved using scipy or the Python Image Library (PIL), image analysis using [https://scikit-image.org/ scikit-image], simple machine-learning using [https://scikit-learn.org/stable/ scikit-learn], etc. Consider also libraries such as [https://github.com/pydata/numexpr NumExpr] to optimize code.
 # '''Re-code''': If code is running slowly, you should identify the bottleneck in the code, and rework it. Typically, using the most appropriate algorithm can improve execution by orders-of-magnitude.
 # '''Don't worry about it''': Before spending serious effort optimizing code for speed, you should decide if it's even necessary. Does it matter if your code takes a minute or an hour to run? Oftentimes it simply isn't worth optimizing code.
@@ Line 26: / Line 26: @@
 #* PyPy ([http://pypy.org/ official site], [https://en.wikipedia.org/wiki/PyPy Wikipedia]) is an alternative Python interpretation, which features JIT. It provides a 2-25&times; speedup. Unfortunately, modules/libraries have to be re-installed/re-compiled into the PyPy environment (separate from the usual Python environment).
 #* Numba ([http://numba.pydata.org/ official site]) uses decorators to allow ultra-easy speedups. (CUDA extension also available.)
 # '''Translation''': There are some attempts to automatically translate Python code into optimized lower-level code.
 #* shedskin ([http://code.google.com/p/shedskin/ official site 1], [http://shedskin.github.io/ official site 2], [https://en.wikipedia.org/wiki/Shed_Skin Wikipedia]) translates Python into C++, providing a 2-200&times; speedup. Most extensions/libraries are not currently supported. On the other hand, one can isolate some critical code and convert this to an optimized external that is called from conventional Python code.

← Older revision		Revision as of 12:39, 25 July 2019
Line 93:		Line 93:

	</source>		</source>
		+
		+	==See Also==
		+	* '''High-performance Python for crystallographic computing'''
		+	** A. Boulle and J. Kieffer [http://scripts.iucr.org/cgi-bin/paper?gj5229 High-performance Python for crystallographic computing] ''J. Appl. Cryst.'' '''2019''', 52 [http://gisaxs.com/index.php?title=Python&action=edit&section=4tps://doi.org/10.1107/S1600576719008471 doi: 10.1107/S1600576719008471]

@@ Line 6: / Line 6: @@
 # '''Make it Pythonic''': Often code runs slowly simply because you are not taking full advantage of Python idioms. Raymond Hettinger has nice notes about "being Pythonic" (e.g. [https://gist.github.com/0x4D31/f0b633548d8e0cfb66ee3bea6a0deff9 notes] or [https://www.youtube.com/watch?feature=player_embedded&v=OSGv2VnC0go video]).
 # '''Libraries''': Exploiting Python libraries (which are highly optimized and often written in lower-level languages) can greatly improve performance. In particular, using numpy for matrix-style numerical computations (rather than using expensive for-loops or other iterations) can massively speedup computations. Processing on images can be improved using the Python Image Library (PIL), etc.
-# '''Re-code''': As always, the first thing to try is to identify the bottleneck in the code, and rework it. Typically, using the most appropriate algorithm can improve execution by orders-of-magnitude.
+# '''Re-code''': If code is running slowly, you should identify the bottleneck in the code, and rework it. Typically, using the most appropriate algorithm can improve execution by orders-of-magnitude.
 # '''Don't worry about it''': Before spending serious effort optimizing code for speed, you should decide if it's even necessary. Does it matter if your code takes a minute or an hour to run? Oftentimes it simply isn't worth optimizing code.
 # '''Parallel''': Python has several mechanisms for adding simple parallelism to your code. For instance if you're processing hundreds of images in sequence, and have a computer with 16 cores, there's no reason you can't load and process 10 images at a time in parallel.

@@ Line 14: / Line 14: @@
 #* multiprocessing ([https://docs.python.org/2/library/multiprocessing.html official docs]) allows one to manage processes.
 #* In some cases you may want to use a more complex workflow system that allows for distributed computation:
-#** [https://dask.org/ dask]
+#** [https://dask.org/ dask] (and [https://github.com/python-streamz/streamz Streamz]) allow one to construct workflows that use multiple workers
 # '''Externals''': Critical code can be written in C/C++, and called as a function within Python. This allows the computational bottleneck to be written in a more specialized and efficient manner.
 #* SWIG ([http://www.swig.org/ official site], [https://en.wikipedia.org/wiki/SWIG Wikipedia]) can provide a 50-200&times; speedup.

← Older revision		Revision as of 12:38, 21 June 2019
Line 1:		Line 1:
−	[[Python]] is a powerful high-level programming language with a clean syntax. However, the flexibility and generality (e.g. dynamic typing) does have an associated performance cost. There are various strategies to improve the speed of execution of Python code:	+	[[Python]] is a powerful high-level programming language with a clean syntax. However, the flexibility and generality (e.g. dynamic typing) does have an associated performance cost.
		+
		+	==Make it faster==
		+	There are various strategies to improve the speed of execution of Python code:

	# '''Make it Pythonic''': Often code runs slowly simply because you are not taking full advantage of Python idioms. Raymond Hettinger has nice notes about "being Pythonic" (e.g. [https://gist.github.com/0x4D31/f0b633548d8e0cfb66ee3bea6a0deff9 notes] or [https://www.youtube.com/watch?feature=player_embedded&v=OSGv2VnC0go video]).		# '''Make it Pythonic''': Often code runs slowly simply because you are not taking full advantage of Python idioms. Raymond Hettinger has nice notes about "being Pythonic" (e.g. [https://gist.github.com/0x4D31/f0b633548d8e0cfb66ee3bea6a0deff9 notes] or [https://www.youtube.com/watch?feature=player_embedded&v=OSGv2VnC0go video]).

@@ Line 63: / Line 63: @@
 print("Slow method timing: {} seconds".format(totslow))
 print("Fast method timing: {} seconds".format(totfast))
 </source>