Skip to content

Instantly share code, notes, and snippets.

@honnibal

honnibal/dynamic_params.py

Last active June 29, 2017 09:30
Show Gist options
  • Save honnibal/050454c8eec1b90c713c4dd65b1cd5a7 to your computer and use it in GitHub Desktop.
Save honnibal/050454c8eec1b90c713c4dd65b1cd5a7 to your computer and use it in GitHub Desktop.
Download ZIP
Cycle hyper parameter
Raw
dynamic_params.py
def cycle_hyper_param(low, high):
'''Dynamically oscillate a hyper-parameter between two values.
Uses the loss momentum to adjust the rate of change. The idea is
that the value should move through regions where the loss is flat
faster, and linger in values where the loss improves.
'''
inc = 0.0001
trend = 0.
prev = 0.
curr = 0.
def get_next_value(new):
nonlocal inc, trend, prev, curr
trend = 0.5 * trend + 0.5 * (prev-new)
curr += inc * (1. / trend)
if curr >= high or curr < low:
inc *= -1
curr = max(low, min(high, curr))
prev = new
return curr
return get_next_value
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment