Skip to content

Instantly share code, notes, and snippets.

@bhrutledge

bhrutledge/testing-notes.md

Last active June 14, 2020 14:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bhrutledge/b6ce92064048a9721771937b8197f8de to your computer and use it in GitHub Desktop.
Save bhrutledge/b6ce92064048a9721771937b8197f8de to your computer and use it in GitHub Desktop.
Download ZIP
Notes from testing articles/talks
Raw
testing-notes.md

Notes from testing articles/talks

Lean Testing or Why Unit Tests are Worse than You Think

  • Collects points from several sources, like Kent Dodds and DHH
  • More focus on integration tests as a sweet spot of confidence/coverage and time
  • Avoid architecture and maintenance overhead of unit tests

Writing Software

David Heinemeier Hansson, RailsConf 2014

  • Committed to building information systems
  • Doesn't align with computer "science"
  • Reading code is like reading old French poetry
  • More like "computer pseudoscience", and diet fads
  • TDD is the most popular software diet
  • "Test at a courser level of granularity", Coplien
  • Unit-test-driven design destroys system architecure
  • System tests are more meaningful
  • Aim for regression tests that allow change w/o breakage
  • Kent Beck: test as little as possible to reach confidence
  • Embrace software "writing" instead of "engineering"
  • Engineering build son hard sciences and metrics
  • Writing focuses on clarity
  • Develop an eye through practice, similar to any creative effort
  • Rewrite drafts for clarity (aka refactor), backed by tests
  • "Delete" key is the most powerful tool

Just Say No to More End-to-End Tests

Google Testing Blog

  • E2E tests are appealing as user simulation
  • But frequently brittle, hard to find root cause
  • To evaluate any testing strategy, evaluate how it enables developers to fix and prevent bugs
  • Feedback loop: fast, reliable, isolates failures
  • Unit tests: a small piece of the product in isolation
  • E2E tests: entire built product; hard to find root cause
  • Integration tests: behavior of small group of units as a whole
  • 70% unit tests, 20% integration tests, and 10% end-to-end tests
  • Avoid inverted pyramid, hourglass

Becoming A Better Programmer

  • Test-driven development != Test-first development
  • "Units" of functionality in isolation
  • "Integration" of units
  • Fully-integrated "system" for "end-to-end" "acceptance"
  • Avoid UI complexity with "subcutaneous" API
  • Test early and often, w/ a short feedback loop
  • Keep tests fast, readable, and maintainable
  • Write failing tests for bugs
  • Test one behavior per test, instead of a specific API

Two Scoops of Django

  • Test one thing: view, model, form
  • Avoid complicated utilities and DRY
  • Test for graceful failure
  • Tests enable smoother upgrades
  • Watch for incremental coverage changes, instead of aiming for 100%
  • Code Coverage Done Right | Codecov

Testing Django applications in 2018

James Bennett, Django core team

  • Use unittest, because that's what Django uses, and has less magic than pytest
  • "Standalone" mode for reusable apps
  • Use tox
  • Write meaningful tests first, then use coverage to detect changes
  • PYTHONWARNINGS=module::DeprecationWarning
  • See https://github.com/ubernostrum/pwned-passwords-django for example

Big picture software testing unit testing, Lean Startup, and everything in between

PyCon 2017

  • Only humans can test for correctness (but automation helps)
  • Automated tests prevent change, for better or worse
  • Define goals and situation to determine testing strategy

The Magic Tricks of Testing

Sandi Metz, Rails Conf 2013

  • We hate our tests because they're slow, fragile, and expensive
  • Unit tests should be thorough, stable, fast, and few
  • Think in terms of objects and messages
    • Query: Return something, change nothing
    • Command: Return nothing, change something
  • Test every part of an object's interface once
  • Trust an object's collaborators
  • Mocks must stay in sync with real API; use libraries to do this automagically
Message Query Command
Incoming Assert result Assert direct public side effects
Self Ignore Ignore
Outgoing Ignore Expect to send

What is this mess? Writing tests for pre-existing code bases

Justin Crown, PyCon 2018

  • Slides
  • No compelling reason not to
  • Test what you touch: start with current, then desired, behavior
  • Favor functional/integration tests
  • Change code cautiously
  • Okay to fix symptoms instead of root cause
  • Write tests to take a break, refresh
  • Mock out network with pytest-socket, etc.
  • TODO: Strategy for new features

Behavior-Driven Python

Andrew Knight, PyCon 2018

Egad! How Do We Start Writing Better Tests?

Andrew Knight, PyGotham 2018

  • Review slides
  • TODO: Screenplay pattern
  • Consider ROI of automation; sometimes manual is okay (but still spec)
  • reportportal.io
  • BDD encourages descriptive, reusable steps
  • BDD helps separate test cases from test code

Fast Test, Slow Test

Gary Bernhardt, PyCon 2012

  • Tests should prevent regression, enable refactoring, and improve design
  • System tests == Functional/Integration/Not-Unit tests
  • System tests don't isolate errors, frequently break multiple at a time
  • System tests prevent regression, but don't enable refactoring or improve design, because they're slow
  • Use system tests for boundaries, unit tests for behavior
  • Aim for 90/10% unit/system tests for object/logic-heavy web apps w/ fewer boundaries
  • Try writing tests from the ground up, by starting with assert
  • Avoid negative assertions, e.g. assert 'foo' not in bar
  • Fail at testing w/ lots of Selenium, big "unit" tests, unit tests for legacy code
  • Unit testing legacy code solidifies it, makes it hard to refactor
  • See "Working Effectively with Legacy Code" for decomposition guidance
  • Consider using service classes for model behaviors, instead of "fat models", to decouple from database

Stop Mocking, Start Testing

Augie Fackler & Nathaniel Manista, PyCon 2012

  • System built of stateless modular components
  • State managed via messages, persisted to database
  • Started w/ poor coverage and manual verification
  • Added tests for new/changed code, but easily bitten by untested code
  • Lots of devs w/ duplicate mocks that got out of sync w/ API
  • Use one well-tested mock (if any)
  • Use inspect (or autospec?) to validate mock signatures
  • Use system tests for gaps in unit tests
  • Use system tests for user stories, not corner cases
  • Aim for a collection of authoritative, narrow, isolated fakes
  • No "has been called X times with Y"
  • TODO: Clarify distinction between fakes, mocks, and stubs
  • Use mocks for services, not business logic
  • Test interface, not implementation
  • Design for love and tests
  • Use explicit dependency injections (instead of defaults)
  • Separate storage from behavior using free functions
  • Don't write unit tests coupled to complex implementations
  • TODO: Tell, don't ask
  • Consider ABC's/interfaces for mocks
  • Write unit tests for behavior and pure functions, not "assembly" code
  • Prefer fakes, not mocks, for code in your control

Using PyTest to Write Beautiful Tests and a Bulletproof Django App

SF Django, Jan 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment