| WORKDIR="/path/for/output/files" | |
| docker run -it --rm -v "${WORKDIR}:/performance/data" -w /performance -- python:3.7 /bin/bash |
To estimate the impact of some changes to dinglehopper I used hyperfine to benchmark the behaviour.
The commands to run a docker container to execute the benchmarks are listed in performance-docker.sh.
The commands needed to prepare, execute and analyse the benchmark are listed in performance.sh.
| # Install hyperfine with dependencies for benchmarking | |
| wget https://github.com/sharkdp/hyperfine/releases/download/v1.11.0/hyperfine_1.11.0_amd64.deb | |
| dpkg -i hyperfine_1.11.0_amd64.deb | |
| git clone --single-branch --branch plt_filenames https://github.com/b2m/hyperfine.git | |
| pip install numpy matplotlib | |
| # clone git repository | |
| git clone https://github.com/b2m/dinglehopper.git | |
| # Setup: commits to check and pairs of GT/OCR-Data | |
| LABELS="before: brochrnx,after: brochnrnx,before: actevedef - tess,after: actevedef - tess,before: actevedef - cal,after: actevedef - cal" | |
| COMMITS=389e253,79ebad2 | |
| DPATH="dinglehopper/qurator/dinglehopper/tests/data" | |
| PAIRS="${DPATH}/brochrnx_73075507X/00000139.gt.page.xml ${DPATH}/brochrnx_73075507X/00000139.ocrd-tess.ocr.page.xml","${DPATH}/actevedef_718448162/OCR-D-GT-PAGE/00000024.page.xml ${DPATH}/actevedef_718448162/OCR-D-OCR-CALAMARI/OCR-D-OCR-CALAMARI_0001.xml","${DPATH}/actevedef_718448162/OCR-D-GT-PAGE/00000024.page.xml ${DPATH}/actevedef_718448162/OCR-D-OCR-TESS/OCR-D-OCR-TESS_0001.xml" | |
| # perform benchmark | |
| hyperfine --parameter-list commit "${COMMITS}" --prepare "cd dinglehopper; git checkout {commit}; pip install .; cd /performance" --warmup 1 --min-runs 5 --parameter-list pair "${PAIRS}" "dinglehopper {pair}" --export-json data/perf-dinglehopper.json --export-markdown data/perf-dinglehopper.md | |
| # data plotting | |
| cd data | |
| python ../hyperfine/scripts/plot_whisker.py --labels="${LABELS}" -- perf-dinglehopper.json |
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
| before brochrnx_73075507X | 3.359 ± 0.039 | 3.305 | 3.408 | 8.29 ± 0.13 |
| after brochrnx_73075507X | 0.405 ± 0.004 | 0.399 | 0.409 | 1.00 |
| before actevedef_718448162 CALAMARI | 34.410 ± 0.561 | 33.918 | 35.362 | 84.91 ± 1.61 |
| after actevedef_718448162 CALAMARI | 0.926 ± 0.010 | 0.911 | 0.935 | 2.28 ± 0.03 |
| before actevedef_718448162 TESS | 34.103 ± 0.305 | 33.685 | 34.529 | 84.16 ± 1.11 |
| after actevedef_718448162 TESS | 0.909 ± 0.008 | 0.899 | 0.921 | 2.24 ± 0.03 |