herbie.tools.FastHerbie#
- class herbie.tools.FastHerbie(DATES, fxx=[0], *, max_threads=50, **kwargs)[source]#
- __init__(DATES, fxx=[0], *, max_threads=50, **kwargs)[source]#
Create many Herbie objects with methods to download or read with xarray.
Uses multithreading.
Note
Currently, Herbie objects looped by run datetime (date) and forecast lead time (fxx).
- Parameters:
DATES (pandas-parsable datetime string or list of datetimes) –
fxx (int or list of forecast lead times) –
max_threads (int) – Maximum number of threads to use.
kwargs – Remaining keywords for Herbie object (e.g., model, product, priority, verbose, etc.)
Benchmark –
--------- –
objects (Creating 48 Herbie) –
1 thread took 16 s
2 threads took 8 s
5 threads took 3.3 s
10 threads took 1.7 s
50 threads took 0.5 s
Methods
__init__
(DATES[, fxx, max_threads])Create many Herbie objects with methods to download or read with xarray.
df
()Organize Herbie objects into a DataFrame.
download
([searchString, max_threads])Download many Herbie objects
xarray
(searchString, *[, max_threads])Read many Herbie objects into an xarray Dataset
Methods:
__init__
(DATES[, fxx, max_threads])Create many Herbie objects with methods to download or read with xarray.
df
()Organize Herbie objects into a DataFrame.
download
([searchString, max_threads])Download many Herbie objects
xarray
(searchString, *[, max_threads])Read many Herbie objects into an xarray Dataset
- __init__(DATES, fxx=[0], *, max_threads=50, **kwargs)[source]#
Create many Herbie objects with methods to download or read with xarray.
Uses multithreading.
Note
Currently, Herbie objects looped by run datetime (date) and forecast lead time (fxx).
- Parameters:
DATES (pandas-parsable datetime string or list of datetimes) –
fxx (int or list of forecast lead times) –
max_threads (int) – Maximum number of threads to use.
kwargs – Remaining keywords for Herbie object (e.g., model, product, priority, verbose, etc.)
Benchmark –
--------- –
objects (Creating 48 Herbie) –
1 thread took 16 s
2 threads took 8 s
5 threads took 3.3 s
10 threads took 1.7 s
50 threads took 0.5 s
- df()[source]#
Organize Herbie objects into a DataFrame.
#? Why is this inefficient? Takes several seconds to display because the __str__ does a lot.
- download(searchString=None, *, max_threads=20, **download_kwargs)[source]#
Download many Herbie objects
Uses multithreading.
- Parameters:
searchString (string) – Regular expression string to specify which GRIB messages to download.
**download_kwargs – Any kwarg for Herbie’s download method.
Benchmark –
--------- –
(TMP (Downloading 48 files with 1 variable) –
1 thread took 1 min 17 s
2 threads took 36 s
5 threads took 28 s
10 threads took 25 s
50 threads took 23 s
- xarray(searchString, *, max_threads=None, **xarray_kwargs)[source]#
Read many Herbie objects into an xarray Dataset
# TODO: Sometimes the Jupyter Cell always crashes when I run this. # TODO: “fatal flex scanner internal error–end of buffer missed”
Uses multithreading (or multiprocessing). This would likely benefit from multiprocessing instead.
- Parameters:
max_threads (int) – Control the maximum number of threads to use. If you use too many threads, you may run into memory limits.
Benchmark –
--------- –
(TMP (Opening 48 files with 1 variable) –
1 thread took 1 min 45 s
2 threads took 55 s
5 threads took 39 s
10 threads took 39 s
50 threads took 37 s