DiskCache API Reference¶
The DiskCache Tutorial provides a helpful walkthrough of most methods.
Cache¶
Read the Cache tutorial for example usage.
- class diskcache.Cache(directory=None, timeout=60, disk=<class 'diskcache.core.Disk'>, **settings)¶
Disk and file backed cache.
- __contains__(key)¶
Return True if key matching item is found in cache.
- Parameters
key – key matching item
- Returns
True if key matching item
- __delitem__(key, retry=True)¶
Delete corresponding item for key from cache.
Raises
Timeout
error when database timeout occurs and retry is False (default True).- Parameters
key – key matching item
retry (bool) – retry if database timeout occurs (default True)
- Raises
KeyError – if key is not found
Timeout – if database timeout occurs
- __getitem__(key)¶
Return corresponding value for key from cache.
- Parameters
key – key matching item
- Returns
corresponding value
- Raises
KeyError – if key is not found
- __init__(directory=None, timeout=60, disk=<class 'diskcache.core.Disk'>, **settings)¶
Initialize cache instance.
- Parameters
directory (str) – cache directory
timeout (float) – SQLite connection timeout
disk – Disk type or subclass for serialization
settings – any of DEFAULT_SETTINGS
- __iter__()¶
Iterate keys in cache including expired items.
- __len__()¶
Count of items in cache including expired items.
- __reversed__()¶
Reverse iterate keys in cache including expired items.
- __setitem__(key, value)¶
Set corresponding value for key in cache.
- Parameters
key – key for item
value – value for item
- Returns
corresponding value
- Raises
KeyError – if key is not found
- add(key, value, expire=None, read=False, tag=None, retry=False)¶
Add key and value item to cache.
Similar to set, but only add to cache if key not present.
Operation is atomic. Only one concurrent add operation for a given key will succeed.
When read is True, value should be a file-like object opened for reading in binary mode.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
key – key for item
value – value for item
expire (float) – seconds until the key expires (default None, no expiry)
read (bool) – read value as bytes from file (default False)
tag (str) – text to associate with key (default None)
retry (bool) – retry if database timeout occurs (default False)
- Returns
True if item was added
- Raises
Timeout – if database timeout occurs
- check(fix=False, retry=False)¶
Check database and file system consistency.
Intended for use in testing and post-mortem error analysis.
While checking the Cache table for consistency, a writer lock is held on the database. The lock blocks other cache clients from writing to the database. For caches with many file references, the lock may be held for a long time. For example, local benchmarking shows that a cache with 1,000 file references takes ~60ms to check.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
fix (bool) – correct inconsistencies
retry (bool) – retry if database timeout occurs (default False)
- Returns
list of warnings
- Raises
Timeout – if database timeout occurs
- clear(retry=False)¶
Remove all items from cache.
Removing items is an iterative process. In each iteration, a subset of items is removed. Concurrent writes may occur between iterations.
If a
Timeout
occurs, the first element of the exception’s args attribute will be the number of items removed before the exception occurred.Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
retry (bool) – retry if database timeout occurs (default False)
- Returns
count of rows removed
- Raises
Timeout – if database timeout occurs
- close()¶
Close database connection.
- create_tag_index()¶
Create tag index on cache database.
It is better to initialize cache with tag_index=True than use this.
- Raises
Timeout – if database timeout occurs
- cull(retry=False)¶
Cull items from cache until volume is less than size limit.
Removing items is an iterative process. In each iteration, a subset of items is removed. Concurrent writes may occur between iterations.
If a
Timeout
occurs, the first element of the exception’s args attribute will be the number of items removed before the exception occurred.Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
retry (bool) – retry if database timeout occurs (default False)
- Returns
count of items removed
- Raises
Timeout – if database timeout occurs
- decr(key, delta=1, default=0, retry=False)¶
Decrement value by delta for item with key.
If key is missing and default is None then raise KeyError. Else if key is missing and default is not None then use default for value.
Operation is atomic. All concurrent decrement operations will be counted individually.
Unlike Memcached, negative values are supported. Value may be decremented below zero.
Assumes value may be stored in a SQLite column. Most builds that target machines with 64-bit pointer widths will support 64-bit signed integers.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
key – key for item
delta (int) – amount to decrement (default 1)
default (int) – value if key is missing (default 0)
retry (bool) – retry if database timeout occurs (default False)
- Returns
new value for item
- Raises
KeyError – if key is not found and default is None
Timeout – if database timeout occurs
- delete(key, retry=False)¶
Delete corresponding item for key from cache.
Missing keys are ignored.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
key – key matching item
retry (bool) – retry if database timeout occurs (default False)
- Returns
True if item was deleted
- Raises
Timeout – if database timeout occurs
- property directory¶
Cache directory.
- property disk¶
Disk used for serialization.
- evict(tag, retry=False)¶
Remove items with matching tag from cache.
Removing items is an iterative process. In each iteration, a subset of items is removed. Concurrent writes may occur between iterations.
If a
Timeout
occurs, the first element of the exception’s args attribute will be the number of items removed before the exception occurred.Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
tag (str) – tag identifying items
retry (bool) – retry if database timeout occurs (default False)
- Returns
count of rows removed
- Raises
Timeout – if database timeout occurs
- expire(now=None, retry=False)¶
Remove expired items from cache.
Removing items is an iterative process. In each iteration, a subset of items is removed. Concurrent writes may occur between iterations.
If a
Timeout
occurs, the first element of the exception’s args attribute will be the number of items removed before the exception occurred.Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
now (float) – current time (default None,
time.time()
used)retry (bool) – retry if database timeout occurs (default False)
- Returns
count of items removed
- Raises
Timeout – if database timeout occurs
- get(key, default=None, read=False, expire_time=False, tag=False, retry=False)¶
Retrieve value from cache. If key is missing, return default.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
key – key for item
default – value to return if key is missing (default None)
read (bool) – if True, return file handle to value (default False)
expire_time (bool) – if True, return expire_time in tuple (default False)
tag (bool) – if True, return tag in tuple (default False)
retry (bool) – retry if database timeout occurs (default False)
- Returns
value for item or default if key not found
- Raises
Timeout – if database timeout occurs
- incr(key, delta=1, default=0, retry=False)¶
Increment value by delta for item with key.
If key is missing and default is None then raise KeyError. Else if key is missing and default is not None then use default for value.
Operation is atomic. All concurrent increment operations will be counted individually.
Assumes value may be stored in a SQLite column. Most builds that target machines with 64-bit pointer widths will support 64-bit signed integers.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
key – key for item
delta (int) – amount to increment (default 1)
default (int) – value if key is missing (default 0)
retry (bool) – retry if database timeout occurs (default False)
- Returns
new value for item
- Raises
KeyError – if key is not found and default is None
Timeout – if database timeout occurs
- iterkeys(reverse=False)¶
Iterate Cache keys in database sort order.
>>> cache = Cache() >>> for key in [4, 1, 3, 0, 2]: ... cache[key] = key >>> list(cache.iterkeys()) [0, 1, 2, 3, 4] >>> list(cache.iterkeys(reverse=True)) [4, 3, 2, 1, 0]
- Parameters
reverse (bool) – reverse sort order (default False)
- Returns
iterator of Cache keys
- memoize(name=None, typed=False, expire=None, tag=None, ignore=())¶
Memoizing cache decorator.
Decorator to wrap callable with memoizing function using cache. Repeated calls with the same arguments will lookup result in cache and avoid function evaluation.
If name is set to None (default), the callable name will be determined automatically.
When expire is set to zero, function results will not be set in the cache. Cache lookups still occur, however. Read Case Study: Landing Page Caching for example usage.
If typed is set to True, function arguments of different types will be cached separately. For example, f(3) and f(3.0) will be treated as distinct calls with distinct results.
The original underlying function is accessible through the __wrapped__ attribute. This is useful for introspection, for bypassing the cache, or for rewrapping the function with a different cache.
>>> from diskcache import Cache >>> cache = Cache() >>> @cache.memoize(expire=1, tag='fib') ... def fibonacci(number): ... if number == 0: ... return 0 ... elif number == 1: ... return 1 ... else: ... return fibonacci(number - 1) + fibonacci(number - 2) >>> print(fibonacci(100)) 354224848179261915075
An additional __cache_key__ attribute can be used to generate the cache key used for the given arguments.
>>> key = fibonacci.__cache_key__(100) >>> print(cache[key]) 354224848179261915075
Remember to call memoize when decorating a callable. If you forget, then a TypeError will occur. Note the lack of parenthenses after memoize below:
>>> @cache.memoize ... def test(): ... pass Traceback (most recent call last): ... TypeError: name cannot be callable
- Parameters
cache – cache to store callable arguments and return values
name (str) – name given for callable (default None, automatic)
typed (bool) – cache different types separately (default False)
expire (float) – seconds until arguments expire (default None, no expiry)
tag (str) – text to associate with arguments (default None)
ignore (set) – positional or keyword args to ignore (default ())
- Returns
callable decorator
- peek(prefix=None, default=(None, None), side='front', expire_time=False, tag=False, retry=False)¶
Peek at key and value item pair from side of queue in cache.
When prefix is None, integer keys are used. Otherwise, string keys are used in the format “prefix-integer”. Integer starts at 500 trillion.
If queue is empty, return default.
Defaults to peeking at key and value item pairs from front of queue. Set side to ‘back’ to pull from back of queue. Side must be one of ‘front’ or ‘back’.
Expired items are deleted from cache. Operation is atomic. Concurrent operations will be serialized.
Raises
Timeout
error when database timeout occurs and retry is False (default).See also Cache.pull and Cache.push.
>>> cache = Cache() >>> for letter in 'abc': ... print(cache.push(letter)) 500000000000000 500000000000001 500000000000002 >>> key, value = cache.peek() >>> print(key) 500000000000000 >>> value 'a' >>> key, value = cache.peek(side='back') >>> print(key) 500000000000002 >>> value 'c'
- Parameters
prefix (str) – key prefix (default None, key is integer)
default – value to return if key is missing (default (None, None))
side (str) – either ‘front’ or ‘back’ (default ‘front’)
expire_time (bool) – if True, return expire_time in tuple (default False)
tag (bool) – if True, return tag in tuple (default False)
retry (bool) – retry if database timeout occurs (default False)
- Returns
key and value item pair or default if queue is empty
- Raises
Timeout – if database timeout occurs
- peekitem(last=True, expire_time=False, tag=False, retry=False)¶
Peek at key and value item pair in cache based on iteration order.
Expired items are deleted from cache. Operation is atomic. Concurrent operations will be serialized.
Raises
Timeout
error when database timeout occurs and retry is False (default).>>> cache = Cache() >>> for num, letter in enumerate('abc'): ... cache[letter] = num >>> cache.peekitem() ('c', 2) >>> cache.peekitem(last=False) ('a', 0)
- Parameters
last (bool) – last item in iteration order (default True)
expire_time (bool) – if True, return expire_time in tuple (default False)
tag (bool) – if True, return tag in tuple (default False)
retry (bool) – retry if database timeout occurs (default False)
- Returns
key and value item pair
- Raises
KeyError – if cache is empty
Timeout – if database timeout occurs
- pop(key, default=None, expire_time=False, tag=False, retry=False)¶
Remove corresponding item for key from cache and return value.
If key is missing, return default.
Operation is atomic. Concurrent operations will be serialized.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
key – key for item
default – value to return if key is missing (default None)
expire_time (bool) – if True, return expire_time in tuple (default False)
tag (bool) – if True, return tag in tuple (default False)
retry (bool) – retry if database timeout occurs (default False)
- Returns
value for item or default if key not found
- Raises
Timeout – if database timeout occurs
- pull(prefix=None, default=(None, None), side='front', expire_time=False, tag=False, retry=False)¶
Pull key and value item pair from side of queue in cache.
When prefix is None, integer keys are used. Otherwise, string keys are used in the format “prefix-integer”. Integer starts at 500 trillion.
If queue is empty, return default.
Defaults to pulling key and value item pairs from front of queue. Set side to ‘back’ to pull from back of queue. Side must be one of ‘front’ or ‘back’.
Operation is atomic. Concurrent operations will be serialized.
Raises
Timeout
error when database timeout occurs and retry is False (default).See also Cache.push and Cache.get.
>>> cache = Cache() >>> cache.pull() (None, None) >>> for letter in 'abc': ... print(cache.push(letter)) 500000000000000 500000000000001 500000000000002 >>> key, value = cache.pull() >>> print(key) 500000000000000 >>> value 'a' >>> _, value = cache.pull(side='back') >>> value 'c' >>> cache.push(1234, 'userids') 'userids-500000000000000' >>> _, value = cache.pull('userids') >>> value 1234
- Parameters
prefix (str) – key prefix (default None, key is integer)
default – value to return if key is missing (default (None, None))
side (str) – either ‘front’ or ‘back’ (default ‘front’)
expire_time (bool) – if True, return expire_time in tuple (default False)
tag (bool) – if True, return tag in tuple (default False)
retry (bool) – retry if database timeout occurs (default False)
- Returns
key and value item pair or default if queue is empty
- Raises
Timeout – if database timeout occurs
- push(value, prefix=None, side='back', expire=None, read=False, tag=None, retry=False)¶
Push value onto side of queue identified by prefix in cache.
When prefix is None, integer keys are used. Otherwise, string keys are used in the format “prefix-integer”. Integer starts at 500 trillion.
Defaults to pushing value on back of queue. Set side to ‘front’ to push value on front of queue. Side must be one of ‘back’ or ‘front’.
Operation is atomic. Concurrent operations will be serialized.
When read is True, value should be a file-like object opened for reading in binary mode.
Raises
Timeout
error when database timeout occurs and retry is False (default).See also Cache.pull.
>>> cache = Cache() >>> print(cache.push('first value')) 500000000000000 >>> cache.get(500000000000000) 'first value' >>> print(cache.push('second value')) 500000000000001 >>> print(cache.push('third value', side='front')) 499999999999999 >>> cache.push(1234, prefix='userids') 'userids-500000000000000'
- Parameters
value – value for item
prefix (str) – key prefix (default None, key is integer)
side (str) – either ‘back’ or ‘front’ (default ‘back’)
expire (float) – seconds until the key expires (default None, no expiry)
read (bool) – read value as bytes from file (default False)
tag (str) – text to associate with key (default None)
retry (bool) – retry if database timeout occurs (default False)
- Returns
key for item in cache
- Raises
Timeout – if database timeout occurs
- read(key, retry=False)¶
Return file handle value corresponding to key from cache.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
key – key matching item
retry (bool) – retry if database timeout occurs (default False)
- Returns
file open for reading in binary mode
- Raises
KeyError – if key is not found
Timeout – if database timeout occurs
- reset(key, value=ENOVAL, update=True)¶
Reset key and value item from Settings table.
Use reset to update the value of Cache settings correctly. Cache settings are stored in the Settings table of the SQLite database. If update is
False
then no attempt is made to update the database.If value is not given, it is reloaded from the Settings table. Otherwise, the Settings table is updated.
Settings with the
disk_
prefix correspond to Disk attributes. Updating the value will change the unprefixed attribute on the associated Disk instance.Settings with the
sqlite_
prefix correspond to SQLite pragmas. Updating the value will execute the corresponding PRAGMA statement.SQLite PRAGMA statements may be executed before the Settings table exists in the database by setting update to
False
.- Parameters
key (str) – Settings key for item
value – value for item (optional)
update (bool) – update database Settings table (default True)
- Returns
updated value for item
- Raises
Timeout – if database timeout occurs
- set(key, value, expire=None, read=False, tag=None, retry=False)¶
Set key and value item in cache.
When read is True, value should be a file-like object opened for reading in binary mode.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
key – key for item
value – value for item
expire (float) – seconds until item expires (default None, no expiry)
read (bool) – read value as bytes from file (default False)
tag (str) – text to associate with key (default None)
retry (bool) – retry if database timeout occurs (default False)
- Returns
True if item was set
- Raises
Timeout – if database timeout occurs
- stats(enable=True, reset=False)¶
Return cache statistics hits and misses.
- Parameters
enable (bool) – enable collecting statistics (default True)
reset (bool) – reset hits and misses to 0 (default False)
- Returns
(hits, misses)
- property timeout¶
SQLite connection timeout value in seconds.
- touch(key, expire=None, retry=False)¶
Touch key in cache and update expire time.
Raises
Timeout
error when database timeout occurs and retry is False (default).- Parameters
key – key for item
expire (float) – seconds until item expires (default None, no expiry)
retry (bool) – retry if database timeout occurs (default False)
- Returns
True if key was touched
- Raises
Timeout – if database timeout occurs
- transact(retry=False)¶
Context manager to perform a transaction by locking the cache.
While the cache is locked, no other write operation is permitted. Transactions should therefore be as short as possible. Read and write operations performed in a transaction are atomic. Read operations may occur concurrent to a transaction.
Transactions may be nested and may not be shared between threads.
Raises
Timeout
error when database timeout occurs and retry is False (default).>>> cache = Cache() >>> with cache.transact(): # Atomically increment two keys. ... _ = cache.incr('total', 123.4) ... _ = cache.incr('count', 1) >>> with cache.transact(): # Atomically calculate average. ... average = cache['total'] / cache['count'] >>> average 123.4
- Parameters
retry (bool) – retry if database timeout occurs (default False)
- Returns
context manager for use in with statement
- Raises
Timeout – if database timeout occurs
- volume()¶
Return estimated total size of cache on disk.
- Returns
size in bytes
FanoutCache¶
Read the FanoutCache tutorial for example usage.
- class diskcache.FanoutCache(directory=None, shards=8, timeout=0.01, disk=<class 'diskcache.core.Disk'>, **settings)¶
Cache that shards keys and values.
- __contains__(key)¶
Return True if key matching item is found in cache.
- Parameters
key – key for item
- Returns
True if key is found
- __delitem__(key)¶
Delete corresponding item for key from cache.
Calls
FanoutCache.delete()
internally with retry set to True.- Parameters
key – key for item
- Raises
KeyError – if key is not found
- __getitem__(key)¶
Return corresponding value for key from cache.
Calls
FanoutCache.get()
internally with retry set to True.- Parameters
key – key for item
- Returns
value for item
- Raises
KeyError – if key is not found
- __init__(directory=None, shards=8, timeout=0.01, disk=<class 'diskcache.core.Disk'>, **settings)¶
Initialize cache instance.
- Parameters
directory (str) – cache directory
shards (int) – number of shards to distribute writes
timeout (float) – SQLite connection timeout
disk – Disk instance for serialization
settings – any of DEFAULT_SETTINGS
- __iter__()¶
Iterate keys in cache including expired items.
- __len__()¶
Count of items in cache including expired items.
- __reversed__()¶
Reverse iterate keys in cache including expired items.
- __setitem__(key, value)¶
Set key and value item in cache.
Calls
FanoutCache.set()
internally with retry set to True.- Parameters
key – key for item
value – value for item
- add(key, value, expire=None, read=False, tag=None, retry=False)¶
Add key and value item to cache.
Similar to set, but only add to cache if key not present.
This operation is atomic. Only one concurrent add operation for given key from separate threads or processes will succeed.
When read is True, value should be a file-like object opened for reading in binary mode.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
key – key for item
value – value for item
expire (float) – seconds until the key expires (default None, no expiry)
read (bool) – read value as bytes from file (default False)
tag (str) – text to associate with key (default None)
retry (bool) – retry if database timeout occurs (default False)
- Returns
True if item was added
- cache(name)¶
Return Cache with given name in subdirectory.
>>> fanout_cache = FanoutCache() >>> cache = fanout_cache.cache('test') >>> cache.set('abc', 123) True >>> cache.get('abc') 123 >>> len(cache) 1 >>> cache.delete('abc') True
- Parameters
name (str) – subdirectory name for Cache
- Returns
Cache with given name
- check(fix=False, retry=False)¶
Check database and file system consistency.
Intended for use in testing and post-mortem error analysis.
While checking the cache table for consistency, a writer lock is held on the database. The lock blocks other cache clients from writing to the database. For caches with many file references, the lock may be held for a long time. For example, local benchmarking shows that a cache with 1,000 file references takes ~60ms to check.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
fix (bool) – correct inconsistencies
retry (bool) – retry if database timeout occurs (default False)
- Returns
list of warnings
- Raises
Timeout – if database timeout occurs
- clear(retry=False)¶
Remove all items from cache.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
retry (bool) – retry if database timeout occurs (default False)
- Returns
count of items removed
- close()¶
Close database connection.
- create_tag_index()¶
Create tag index on cache database.
Better to initialize cache with tag_index=True than use this.
- Raises
Timeout – if database timeout occurs
- cull(retry=False)¶
Cull items from cache until volume is less than size limit.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
retry (bool) – retry if database timeout occurs (default False)
- Returns
count of items removed
- decr(key, delta=1, default=0, retry=False)¶
Decrement value by delta for item with key.
If key is missing and default is None then raise KeyError. Else if key is missing and default is not None then use default for value.
Operation is atomic. All concurrent decrement operations will be counted individually.
Unlike Memcached, negative values are supported. Value may be decremented below zero.
Assumes value may be stored in a SQLite column. Most builds that target machines with 64-bit pointer widths will support 64-bit signed integers.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
key – key for item
delta (int) – amount to decrement (default 1)
default (int) – value if key is missing (default 0)
retry (bool) – retry if database timeout occurs (default False)
- Returns
new value for item on success else None
- Raises
KeyError – if key is not found and default is None
- delete(key, retry=False)¶
Delete corresponding item for key from cache.
Missing keys are ignored.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
key – key for item
retry (bool) – retry if database timeout occurs (default False)
- Returns
True if item was deleted
- deque(name)¶
Return Deque with given name in subdirectory.
>>> cache = FanoutCache() >>> deque = cache.deque('test') >>> deque.extend('abc') >>> deque.popleft() 'a' >>> deque.pop() 'c' >>> len(deque) 1
- Parameters
name (str) – subdirectory name for Deque
- Returns
Deque with given name
- property directory¶
Cache directory.
- evict(tag, retry=False)¶
Remove items with matching tag from cache.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
tag (str) – tag identifying items
retry (bool) – retry if database timeout occurs (default False)
- Returns
count of items removed
- expire(retry=False)¶
Remove expired items from cache.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
retry (bool) – retry if database timeout occurs (default False)
- Returns
count of items removed
- get(key, default=None, read=False, expire_time=False, tag=False, retry=False)¶
Retrieve value from cache. If key is missing, return default.
If database timeout occurs then returns default unless retry is set to True (default False).
- Parameters
key – key for item
default – return value if key is missing (default None)
read (bool) – if True, return file handle to value (default False)
expire_time (float) – if True, return expire_time in tuple (default False)
tag – if True, return tag in tuple (default False)
retry (bool) – retry if database timeout occurs (default False)
- Returns
value for item if key is found else default
- incr(key, delta=1, default=0, retry=False)¶
Increment value by delta for item with key.
If key is missing and default is None then raise KeyError. Else if key is missing and default is not None then use default for value.
Operation is atomic. All concurrent increment operations will be counted individually.
Assumes value may be stored in a SQLite column. Most builds that target machines with 64-bit pointer widths will support 64-bit signed integers.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
key – key for item
delta (int) – amount to increment (default 1)
default (int) – value if key is missing (default 0)
retry (bool) – retry if database timeout occurs (default False)
- Returns
new value for item on success else None
- Raises
KeyError – if key is not found and default is None
- index(name)¶
Return Index with given name in subdirectory.
>>> cache = FanoutCache() >>> index = cache.index('test') >>> index['abc'] = 123 >>> index['def'] = 456 >>> index['ghi'] = 789 >>> index.popitem() ('ghi', 789) >>> del index['abc'] >>> len(index) 1 >>> index['def'] 456
- Parameters
name (str) – subdirectory name for Index
- Returns
Index with given name
- memoize(name=None, typed=False, expire=None, tag=None, ignore=())¶
Memoizing cache decorator.
Decorator to wrap callable with memoizing function using cache. Repeated calls with the same arguments will lookup result in cache and avoid function evaluation.
If name is set to None (default), the callable name will be determined automatically.
When expire is set to zero, function results will not be set in the cache. Cache lookups still occur, however. Read Case Study: Landing Page Caching for example usage.
If typed is set to True, function arguments of different types will be cached separately. For example, f(3) and f(3.0) will be treated as distinct calls with distinct results.
The original underlying function is accessible through the __wrapped__ attribute. This is useful for introspection, for bypassing the cache, or for rewrapping the function with a different cache.
>>> from diskcache import Cache >>> cache = Cache() >>> @cache.memoize(expire=1, tag='fib') ... def fibonacci(number): ... if number == 0: ... return 0 ... elif number == 1: ... return 1 ... else: ... return fibonacci(number - 1) + fibonacci(number - 2) >>> print(fibonacci(100)) 354224848179261915075
An additional __cache_key__ attribute can be used to generate the cache key used for the given arguments.
>>> key = fibonacci.__cache_key__(100) >>> print(cache[key]) 354224848179261915075
Remember to call memoize when decorating a callable. If you forget, then a TypeError will occur. Note the lack of parenthenses after memoize below:
>>> @cache.memoize ... def test(): ... pass Traceback (most recent call last): ... TypeError: name cannot be callable
- Parameters
cache – cache to store callable arguments and return values
name (str) – name given for callable (default None, automatic)
typed (bool) – cache different types separately (default False)
expire (float) – seconds until arguments expire (default None, no expiry)
tag (str) – text to associate with arguments (default None)
ignore (set) – positional or keyword args to ignore (default ())
- Returns
callable decorator
- pop(key, default=None, expire_time=False, tag=False, retry=False)¶
Remove corresponding item for key from cache and return value.
If key is missing, return default.
Operation is atomic. Concurrent operations will be serialized.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
key – key for item
default – return value if key is missing (default None)
expire_time (float) – if True, return expire_time in tuple (default False)
tag – if True, return tag in tuple (default False)
retry (bool) – retry if database timeout occurs (default False)
- Returns
value for item if key is found else default
- read(key)¶
Return file handle corresponding to key from cache.
- Parameters
key – key for item
- Returns
file open for reading in binary mode
- Raises
KeyError – if key is not found
- reset(key, value=ENOVAL)¶
Reset key and value item from Settings table.
If value is not given, it is reloaded from the Settings table. Otherwise, the Settings table is updated.
Settings attributes on cache objects are lazy-loaded and read-only. Use reset to update the value.
Settings with the
sqlite_
prefix correspond to SQLite pragmas. Updating the value will execute the corresponding PRAGMA statement.- Parameters
key (str) – Settings key for item
value – value for item (optional)
- Returns
updated value for item
- set(key, value, expire=None, read=False, tag=None, retry=False)¶
Set key and value item in cache.
When read is True, value should be a file-like object opened for reading in binary mode.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
key – key for item
value – value for item
expire (float) – seconds until the key expires (default None, no expiry)
read (bool) – read value as raw bytes from file (default False)
tag (str) – text to associate with key (default None)
retry (bool) – retry if database timeout occurs (default False)
- Returns
True if item was set
- stats(enable=True, reset=False)¶
Return cache statistics hits and misses.
- Parameters
enable (bool) – enable collecting statistics (default True)
reset (bool) – reset hits and misses to 0 (default False)
- Returns
(hits, misses)
- touch(key, expire=None, retry=False)¶
Touch key in cache and update expire time.
If database timeout occurs then fails silently unless retry is set to True (default False).
- Parameters
key – key for item
expire (float) – seconds until the key expires (default None, no expiry)
retry (bool) – retry if database timeout occurs (default False)
- Returns
True if key was touched
- transact(retry=True)¶
Context manager to perform a transaction by locking the cache.
While the cache is locked, no other write operation is permitted. Transactions should therefore be as short as possible. Read and write operations performed in a transaction are atomic. Read operations may occur concurrent to a transaction.
Transactions may be nested and may not be shared between threads.
Blocks until transactions are held on all cache shards by retrying as necessary.
>>> cache = FanoutCache() >>> with cache.transact(): # Atomically increment two keys. ... _ = cache.incr('total', 123.4) ... _ = cache.incr('count', 1) >>> with cache.transact(): # Atomically calculate average. ... average = cache['total'] / cache['count'] >>> average 123.4
- Returns
context manager for use in with statement
- volume()¶
Return estimated total size of cache on disk.
- Returns
size in bytes
DjangoCache¶
Read the DjangoCache tutorial for example usage.
- class diskcache.DjangoCache(directory, params)¶
Django-compatible disk and file backed cache.
- __init__(directory, params)¶
Initialize DjangoCache instance.
- Parameters
directory (str) – cache directory
params (dict) – cache parameters
- add(key, value, timeout=<object object>, version=None, read=False, tag=None, retry=True)¶
Set a value in the cache if the key does not already exist. If timeout is given, that timeout will be used for the key; otherwise the default cache timeout will be used.
Return True if the value was stored, False otherwise.
- Parameters
key – key for item
value – value for item
timeout (float) – seconds until the item expires (default 300 seconds)
version (int) – key version number (default None, cache parameter)
read (bool) – read value as bytes from file (default False)
tag (str) – text to associate with key (default None)
retry (bool) – retry if database timeout occurs (default True)
- Returns
True if item was added
- cache(name)¶
Return Cache with given name in subdirectory.
- Parameters
name (str) – subdirectory name for Cache
- Returns
Cache with given name
- clear()¶
Remove all values from the cache at once.
- close(**kwargs)¶
Close the cache connection.
- create_tag_index()¶
Create tag index on cache database.
Better to initialize cache with tag_index=True than use this.
- Raises
Timeout – if database timeout occurs
- cull()¶
Cull items from cache until volume is less than size limit.
- Returns
count of items removed
- decr(key, delta=1, version=None, default=None, retry=True)¶
Decrement value by delta for item with key.
If key is missing and default is None then raise KeyError. Else if key is missing and default is not None then use default for value.
Operation is atomic. All concurrent decrement operations will be counted individually.
Unlike Memcached, negative values are supported. Value may be decremented below zero.
Assumes value may be stored in a SQLite column. Most builds that target machines with 64-bit pointer widths will support 64-bit signed integers.
- Parameters
key – key for item
delta (int) – amount to decrement (default 1)
version (int) – key version number (default None, cache parameter)
default (int) – value if key is missing (default None)
retry (bool) – retry if database timeout occurs (default True)
- Returns
new value for item on success else None
- Raises
ValueError – if key is not found and default is None
- delete(key, version=None, retry=True)¶
Delete a key from the cache, failing silently.
- Parameters
key – key for item
version (int) – key version number (default None, cache parameter)
retry (bool) – retry if database timeout occurs (default True)
- Returns
True if item was deleted
- deque(name)¶
Return Deque with given name in subdirectory.
- Parameters
name (str) – subdirectory name for Deque
- Returns
Deque with given name
- property directory¶
Cache directory.
- evict(tag)¶
Remove items with matching tag from cache.
- Parameters
tag (str) – tag identifying items
- Returns
count of items removed
- expire()¶
Remove expired items from cache.
- Returns
count of items removed
- get(key, default=None, version=None, read=False, expire_time=False, tag=False, retry=False)¶
Fetch a given key from the cache. If the key does not exist, return default, which itself defaults to None.
- Parameters
key – key for item
default – return value if key is missing (default None)
version (int) – key version number (default None, cache parameter)
read (bool) – if True, return file handle to value (default False)
expire_time (float) – if True, return expire_time in tuple (default False)
tag – if True, return tag in tuple (default False)
retry (bool) – retry if database timeout occurs (default False)
- Returns
value for item if key is found else default
- get_backend_timeout(timeout=<object object>)¶
Return seconds to expiration.
- Parameters
timeout (float) – seconds until the item expires (default 300 seconds)
- has_key(key, version=None)¶
Returns True if the key is in the cache and has not expired.
- Parameters
key – key for item
version (int) – key version number (default None, cache parameter)
- Returns
True if key is found
- incr(key, delta=1, version=None, default=None, retry=True)¶
Increment value by delta for item with key.
If key is missing and default is None then raise KeyError. Else if key is missing and default is not None then use default for value.
Operation is atomic. All concurrent increment operations will be counted individually.
Assumes value may be stored in a SQLite column. Most builds that target machines with 64-bit pointer widths will support 64-bit signed integers.
- Parameters
key – key for item
delta (int) – amount to increment (default 1)
version (int) – key version number (default None, cache parameter)
default (int) – value if key is missing (default None)
retry (bool) – retry if database timeout occurs (default True)
- Returns
new value for item on success else None
- Raises
ValueError – if key is not found and default is None
- index(name)¶
Return Index with given name in subdirectory.
- Parameters
name (str) – subdirectory name for Index
- Returns
Index with given name
- memoize(name=None, timeout=<object object>, version=None, typed=False, tag=None, ignore=())¶
Memoizing cache decorator.
Decorator to wrap callable with memoizing function using cache. Repeated calls with the same arguments will lookup result in cache and avoid function evaluation.
If name is set to None (default), the callable name will be determined automatically.
When timeout is set to zero, function results will not be set in the cache. Cache lookups still occur, however. Read Case Study: Landing Page Caching for example usage.
If typed is set to True, function arguments of different types will be cached separately. For example, f(3) and f(3.0) will be treated as distinct calls with distinct results.
The original underlying function is accessible through the __wrapped__ attribute. This is useful for introspection, for bypassing the cache, or for rewrapping the function with a different cache.
An additional __cache_key__ attribute can be used to generate the cache key used for the given arguments.
Remember to call memoize when decorating a callable. If you forget, then a TypeError will occur.
- Parameters
name (str) – name given for callable (default None, automatic)
timeout (float) – seconds until the item expires (default 300 seconds)
version (int) – key version number (default None, cache parameter)
typed (bool) – cache different types separately (default False)
tag (str) – text to associate with arguments (default None)
ignore (set) – positional or keyword args to ignore (default ())
- Returns
callable decorator
- pop(key, default=None, version=None, expire_time=False, tag=False, retry=True)¶
Remove corresponding item for key from cache and return value.
If key is missing, return default.
Operation is atomic. Concurrent operations will be serialized.
- Parameters
key – key for item
default – return value if key is missing (default None)
version (int) – key version number (default None, cache parameter)
expire_time (float) – if True, return expire_time in tuple (default False)
tag – if True, return tag in tuple (default False)
retry (bool) – retry if database timeout occurs (default True)
- Returns
value for item if key is found else default
- read(key, version=None)¶
Return file handle corresponding to key from Cache.
- Parameters
key – Python key to retrieve
version (int) – key version number (default None, cache parameter)
- Returns
file open for reading in binary mode
- Raises
KeyError – if key is not found
- set(key, value, timeout=<object object>, version=None, read=False, tag=None, retry=True)¶
Set a value in the cache. If timeout is given, that timeout will be used for the key; otherwise the default cache timeout will be used.
- Parameters
key – key for item
value – value for item
timeout (float) – seconds until the item expires (default 300 seconds)
version (int) – key version number (default None, cache parameter)
read (bool) – read value as bytes from file (default False)
tag (str) – text to associate with key (default None)
retry (bool) – retry if database timeout occurs (default True)
- Returns
True if item was set
- stats(enable=True, reset=False)¶
Return cache statistics hits and misses.
- Parameters
enable (bool) – enable collecting statistics (default True)
reset (bool) – reset hits and misses to 0 (default False)
- Returns
(hits, misses)
- touch(key, timeout=<object object>, version=None, retry=True)¶
Touch a key in the cache. If timeout is given, that timeout will be used for the key; otherwise the default cache timeout will be used.
- Parameters
key – key for item
timeout (float) – seconds until the item expires (default 300 seconds)
version (int) – key version number (default None, cache parameter)
retry (bool) – retry if database timeout occurs (default True)
- Returns
True if key was touched
Deque¶
- class diskcache.Deque(iterable=(), directory=None)¶
Persistent sequence with double-ended queue semantics.
Double-ended queue is an ordered collection with optimized access at its endpoints.
Items are serialized to disk. Deque may be initialized from directory path where items are stored.
>>> deque = Deque() >>> deque += range(5) >>> list(deque) [0, 1, 2, 3, 4] >>> for value in range(5): ... deque.appendleft(-value) >>> len(deque) 10 >>> list(deque) [-4, -3, -2, -1, 0, 0, 1, 2, 3, 4] >>> deque.pop() 4 >>> deque.popleft() -4 >>> deque.reverse() >>> list(deque) [3, 2, 1, 0, 0, -1, -2, -3]
- __delitem__(index)¶
deque.__delitem__(index) <==> del deque[index]
Delete item in deque at index.
>>> deque = Deque() >>> deque.extend([None] * 3) >>> del deque[0] >>> del deque[1] >>> del deque[-1] >>> len(deque) 0
- Parameters
index (int) – index of item
- Raises
IndexError – if index out of range
- __eq__(that)¶
Return True if and only if deque is equal to that.
- __ge__(that)¶
Return True if and only if deque is greater than or equal to that.
- __getitem__(index)¶
deque.__getitem__(index) <==> deque[index]
Return corresponding item for index in deque.
See also Deque.peekleft and Deque.peek for indexing deque at index
0
or-1
.>>> deque = Deque() >>> deque.extend('abcde') >>> deque[1] 'b' >>> deque[-2] 'd'
- Parameters
index (int) – index of item
- Returns
corresponding item
- Raises
IndexError – if index out of range
- __gt__(that)¶
Return True if and only if deque is greater than that.
- __hash__ = None¶
- __iadd__(iterable)¶
deque.__iadd__(iterable) <==> deque += iterable
Extend back side of deque with items from iterable.
- Parameters
iterable – iterable of items to append to deque
- Returns
deque with added items
- __init__(iterable=(), directory=None)¶
Initialize deque instance.
If directory is None then temporary directory created. The directory will not be automatically removed.
- Parameters
iterable – iterable of items to append to deque
directory – deque directory (default None)
- __iter__() <==> iter(deque)¶
Return iterator of deque from front to back.
- __le__(that)¶
Return True if and only if deque is less than or equal to that.
- __len__() <==> len(deque)¶
Return length of deque.
- __lt__(that)¶
Return True if and only if deque is less than that.
- __ne__(that)¶
Return True if and only if deque is not equal to that.
- __repr__() <==> repr(deque)¶
Return string with printable representation of deque.
- __reversed__() <==> reversed(deque)¶
Return iterator of deque from back to front.
>>> deque = Deque() >>> deque.extend('abcd') >>> iterator = reversed(deque) >>> next(iterator) 'd' >>> list(iterator) ['c', 'b', 'a']
- __setitem__(index, value)¶
deque.__setitem__(index, value) <==> deque[index] = value
Store value in deque at index.
>>> deque = Deque() >>> deque.extend([None] * 3) >>> deque[0] = 'a' >>> deque[1] = 'b' >>> deque[-1] = 'c' >>> ''.join(deque) 'abc'
- Parameters
index (int) – index of value
value – value to store
- Raises
IndexError – if index out of range
- append(value)¶
Add value to back of deque.
>>> deque = Deque() >>> deque.append('a') >>> deque.append('b') >>> deque.append('c') >>> list(deque) ['a', 'b', 'c']
- Parameters
value – value to add to back of deque
- appendleft(value)¶
Add value to front of deque.
>>> deque = Deque() >>> deque.appendleft('a') >>> deque.appendleft('b') >>> deque.appendleft('c') >>> list(deque) ['c', 'b', 'a']
- Parameters
value – value to add to front of deque
- property cache¶
Cache used by deque.
- clear()¶
Remove all elements from deque.
>>> deque = Deque('abc') >>> len(deque) 3 >>> deque.clear() >>> list(deque) []
- count(value)¶
Return number of occurrences of value in deque.
>>> deque = Deque() >>> deque += [num for num in range(1, 5) for _ in range(num)] >>> deque.count(0) 0 >>> deque.count(1) 1 >>> deque.count(4) 4
- Parameters
value – value to count in deque
- Returns
count of items equal to value in deque
- property directory¶
Directory path where deque is stored.
- extend(iterable)¶
Extend back side of deque with values from iterable.
- Parameters
iterable – iterable of values
- extendleft(iterable)¶
Extend front side of deque with value from iterable.
>>> deque = Deque() >>> deque.extendleft('abc') >>> list(deque) ['c', 'b', 'a']
- Parameters
iterable – iterable of values
- classmethod fromcache(cache, iterable=())¶
Initialize deque using cache.
>>> cache = Cache() >>> deque = Deque.fromcache(cache, [5, 6, 7, 8]) >>> deque.cache is cache True >>> len(deque) 4 >>> 7 in deque True >>> deque.popleft() 5
- Parameters
cache (Cache) – cache to use
iterable – iterable of items
- Returns
initialized Deque
- peek()¶
Peek at value at back of deque.
Faster than indexing deque at -1.
If deque is empty then raise IndexError.
>>> deque = Deque() >>> deque.peek() Traceback (most recent call last): ... IndexError: peek from an empty deque >>> deque += 'abc' >>> deque.peek() 'c'
- Returns
value at back of deque
- Raises
IndexError – if deque is empty
- peekleft()¶
Peek at value at front of deque.
Faster than indexing deque at 0.
If deque is empty then raise IndexError.
>>> deque = Deque() >>> deque.peekleft() Traceback (most recent call last): ... IndexError: peek from an empty deque >>> deque += 'abc' >>> deque.peekleft() 'a'
- Returns
value at front of deque
- Raises
IndexError – if deque is empty
- pop()¶
Remove and return value at back of deque.
If deque is empty then raise IndexError.
>>> deque = Deque() >>> deque += 'ab' >>> deque.pop() 'b' >>> deque.pop() 'a' >>> deque.pop() Traceback (most recent call last): ... IndexError: pop from an empty deque
- Returns
value at back of deque
- Raises
IndexError – if deque is empty
- popleft()¶
Remove and return value at front of deque.
>>> deque = Deque() >>> deque += 'ab' >>> deque.popleft() 'a' >>> deque.popleft() 'b' >>> deque.popleft() Traceback (most recent call last): ... IndexError: pop from an empty deque
- Returns
value at front of deque
- Raises
IndexError – if deque is empty
- remove(value)¶
Remove first occurrence of value in deque.
>>> deque = Deque() >>> deque += 'aab' >>> deque.remove('a') >>> list(deque) ['a', 'b'] >>> deque.remove('b') >>> list(deque) ['a'] >>> deque.remove('c') Traceback (most recent call last): ... ValueError: deque.remove(value): value not in deque
- Parameters
value – value to remove
- Raises
ValueError – if value not in deque
- reverse()¶
Reverse deque in place.
>>> deque = Deque() >>> deque += 'abc' >>> deque.reverse() >>> list(deque) ['c', 'b', 'a']
- rotate(steps=1)¶
Rotate deque right by steps.
If steps is negative then rotate left.
>>> deque = Deque() >>> deque += range(5) >>> deque.rotate(2) >>> list(deque) [3, 4, 0, 1, 2] >>> deque.rotate(-1) >>> list(deque) [4, 0, 1, 2, 3]
- Parameters
steps (int) – number of steps to rotate (default 1)
- transact()¶
Context manager to perform a transaction by locking the deque.
While the deque is locked, no other write operation is permitted. Transactions should therefore be as short as possible. Read and write operations performed in a transaction are atomic. Read operations may occur concurrent to a transaction.
Transactions may be nested and may not be shared between threads.
>>> from diskcache import Deque >>> deque = Deque() >>> deque += range(5) >>> with deque.transact(): # Atomically rotate elements. ... value = deque.pop() ... deque.appendleft(value) >>> list(deque) [4, 0, 1, 2, 3]
- Returns
context manager for use in with statement
Index¶
- class diskcache.Index(*args, **kwargs)¶
Persistent mutable mapping with insertion order iteration.
Items are serialized to disk. Index may be initialized from directory path where items are stored.
Hashing protocol is not used. Keys are looked up by their serialized format. See
diskcache.Disk
for details.>>> index = Index() >>> index.update([('a', 1), ('b', 2), ('c', 3)]) >>> index['a'] 1 >>> list(index) ['a', 'b', 'c'] >>> len(index) 3 >>> del index['b'] >>> index.popitem() ('c', 3)
- __delitem__(key)¶
index.__delitem__(key) <==> del index[key]
Delete corresponding item for key from index.
>>> index = Index() >>> index.update({'a': 1, 'b': 2}) >>> del index['a'] >>> del index['b'] >>> len(index) 0 >>> del index['c'] Traceback (most recent call last): ... KeyError: 'c'
- Parameters
key – key for item
- Raises
KeyError – if key is not found
- __eq__(other)¶
index.__eq__(other) <==> index == other
Compare equality for index and other.
Comparison to another index or ordered dictionary is order-sensitive. Comparison to all other mappings is order-insensitive.
>>> index = Index() >>> pairs = [('a', 1), ('b', 2), ('c', 3)] >>> index.update(pairs) >>> from collections import OrderedDict >>> od = OrderedDict(pairs) >>> index == od True >>> index == {'c': 3, 'b': 2, 'a': 1} True
- Parameters
other – other mapping in equality comparison
- Returns
True if index equals other
- __getitem__(key)¶
index.__getitem__(key) <==> index[key]
Return corresponding value for key in index.
>>> index = Index() >>> index.update({'a': 1, 'b': 2}) >>> index['a'] 1 >>> index['b'] 2 >>> index['c'] Traceback (most recent call last): ... KeyError: 'c'
- Parameters
key – key for item
- Returns
value for item in index with given key
- Raises
KeyError – if key is not found
- __hash__ = None¶
- __init__(*args, **kwargs)¶
Initialize index in directory and update items.
Optional first argument may be string specifying directory where items are stored. When None or not given, temporary directory is created.
>>> index = Index({'a': 1, 'b': 2, 'c': 3}) >>> len(index) 3 >>> directory = index.directory >>> inventory = Index(directory, d=4) >>> inventory['b'] 2 >>> len(inventory) 4
- __iter__() <==> iter(index)¶
Return iterator of index keys in insertion order.
- __len__() <==> len(index)¶
Return length of index.
- __ne__(other)¶
index.__ne__(other) <==> index != other
Compare inequality for index and other.
Comparison to another index or ordered dictionary is order-sensitive. Comparison to all other mappings is order-insensitive.
>>> index = Index() >>> index.update([('a', 1), ('b', 2), ('c', 3)]) >>> from collections import OrderedDict >>> od = OrderedDict([('c', 3), ('b', 2), ('a', 1)]) >>> index != od True >>> index != {'a': 1, 'b': 2} True
- Parameters
other – other mapping in inequality comparison
- Returns
True if index does not equal other
- __repr__() <==> repr(index)¶
Return string with printable representation of index.
- __reversed__() <==> reversed(index)¶
Return iterator of index keys in reversed insertion order.
>>> index = Index() >>> index.update([('a', 1), ('b', 2), ('c', 3)]) >>> iterator = reversed(index) >>> next(iterator) 'c' >>> list(iterator) ['b', 'a']
- __setitem__(key, value)¶
index.__setitem__(key, value) <==> index[key] = value
Set key and value item in index.
>>> index = Index() >>> index['a'] = 1 >>> index[0] = None >>> len(index) 2
- Parameters
key – key for item
value – value for item
- property cache¶
Cache used by index.
- clear()¶
Remove all items from index.
>>> index = Index({'a': 0, 'b': 1, 'c': 2}) >>> len(index) 3 >>> index.clear() >>> dict(index) {}
- property directory¶
Directory path where items are stored.
- classmethod fromcache(cache, *args, **kwargs)¶
Initialize index using cache and update items.
>>> cache = Cache() >>> index = Index.fromcache(cache, {'a': 1, 'b': 2, 'c': 3}) >>> index.cache is cache True >>> len(index) 3 >>> 'b' in index True >>> index['c'] 3
- Parameters
cache (Cache) – cache to use
args – mapping or sequence of items
kwargs – mapping of items
- Returns
initialized Index
- items()¶
Set-like object providing a view of index items.
>>> index = Index() >>> index.update({'a': 1, 'b': 2, 'c': 3}) >>> items_view = index.items() >>> ('b', 2) in items_view True
- Returns
items view
- keys()¶
Set-like object providing a view of index keys.
>>> index = Index() >>> index.update({'a': 1, 'b': 2, 'c': 3}) >>> keys_view = index.keys() >>> 'b' in keys_view True
- Returns
keys view
- memoize(name=None, typed=False, ignore=())¶
Memoizing cache decorator.
Decorator to wrap callable with memoizing function using cache. Repeated calls with the same arguments will lookup result in cache and avoid function evaluation.
If name is set to None (default), the callable name will be determined automatically.
If typed is set to True, function arguments of different types will be cached separately. For example, f(3) and f(3.0) will be treated as distinct calls with distinct results.
The original underlying function is accessible through the __wrapped__ attribute. This is useful for introspection, for bypassing the cache, or for rewrapping the function with a different cache.
>>> from diskcache import Index >>> mapping = Index() >>> @mapping.memoize() ... def fibonacci(number): ... if number == 0: ... return 0 ... elif number == 1: ... return 1 ... else: ... return fibonacci(number - 1) + fibonacci(number - 2) >>> print(fibonacci(100)) 354224848179261915075
An additional __cache_key__ attribute can be used to generate the cache key used for the given arguments.
>>> key = fibonacci.__cache_key__(100) >>> print(mapping[key]) 354224848179261915075
Remember to call memoize when decorating a callable. If you forget, then a TypeError will occur. Note the lack of parenthenses after memoize below:
>>> @mapping.memoize ... def test(): ... pass Traceback (most recent call last): ... TypeError: name cannot be callable
- Parameters
name (str) – name given for callable (default None, automatic)
typed (bool) – cache different types separately (default False)
ignore (set) – positional or keyword args to ignore (default ())
- Returns
callable decorator
- peekitem(last=True)¶
Peek at key and value item pair in index based on iteration order.
>>> index = Index() >>> for num, letter in enumerate('xyz'): ... index[letter] = num >>> index.peekitem() ('z', 2) >>> index.peekitem(last=False) ('x', 0)
- Parameters
last (bool) – last item in iteration order (default True)
- Returns
key and value item pair
- Raises
KeyError – if cache is empty
- pop(key, default=ENOVAL)¶
Remove corresponding item for key from index and return value.
If key is missing then return default. If default is ENOVAL then raise KeyError.
>>> index = Index({'a': 1, 'b': 2}) >>> index.pop('a') 1 >>> index.pop('b') 2 >>> index.pop('c', default=3) 3 >>> index.pop('d') Traceback (most recent call last): ... KeyError: 'd'
- Parameters
key – key for item
default – return value if key is missing (default ENOVAL)
- Returns
value for item if key is found else default
- Raises
KeyError – if key is not found and default is ENOVAL
- popitem(last=True)¶
Remove and return item pair.
Item pairs are returned in last-in-first-out (LIFO) order if last is True else first-in-first-out (FIFO) order. LIFO order imitates a stack and FIFO order imitates a queue.
>>> index = Index() >>> index.update([('a', 1), ('b', 2), ('c', 3)]) >>> index.popitem() ('c', 3) >>> index.popitem(last=False) ('a', 1) >>> index.popitem() ('b', 2) >>> index.popitem() Traceback (most recent call last): ... KeyError: 'dictionary is empty'
- Parameters
last (bool) – pop last item pair (default True)
- Returns
key and value item pair
- Raises
KeyError – if index is empty
- pull(prefix=None, default=(None, None), side='front')¶
Pull key and value item pair from side of queue in index.
When prefix is None, integer keys are used. Otherwise, string keys are used in the format “prefix-integer”. Integer starts at 500 trillion.
If queue is empty, return default.
Defaults to pulling key and value item pairs from front of queue. Set side to ‘back’ to pull from back of queue. Side must be one of ‘front’ or ‘back’.
See also Index.push.
>>> index = Index() >>> for letter in 'abc': ... print(index.push(letter)) 500000000000000 500000000000001 500000000000002 >>> key, value = index.pull() >>> print(key) 500000000000000 >>> value 'a' >>> _, value = index.pull(side='back') >>> value 'c' >>> index.pull(prefix='fruit') (None, None)
- Parameters
prefix (str) – key prefix (default None, key is integer)
default – value to return if key is missing (default (None, None))
side (str) – either ‘front’ or ‘back’ (default ‘front’)
- Returns
key and value item pair or default if queue is empty
- push(value, prefix=None, side='back')¶
Push value onto side of queue in index identified by prefix.
When prefix is None, integer keys are used. Otherwise, string keys are used in the format “prefix-integer”. Integer starts at 500 trillion.
Defaults to pushing value on back of queue. Set side to ‘front’ to push value on front of queue. Side must be one of ‘back’ or ‘front’.
See also Index.pull.
>>> index = Index() >>> print(index.push('apples')) 500000000000000 >>> print(index.push('beans')) 500000000000001 >>> print(index.push('cherries', side='front')) 499999999999999 >>> index[500000000000001] 'beans' >>> index.push('dates', prefix='fruit') 'fruit-500000000000000'
- Parameters
value – value for item
prefix (str) – key prefix (default None, key is integer)
side (str) – either ‘back’ or ‘front’ (default ‘back’)
- Returns
key for item in cache
- setdefault(key, default=None)¶
Set and get value for key in index using default.
If key is not in index then set corresponding value to default. If key is in index then ignore default and return existing value.
>>> index = Index() >>> index.setdefault('a', 0) 0 >>> index.setdefault('a', 1) 0
- Parameters
key – key for item
default – value if key is missing (default None)
- Returns
value for item in index with given key
- transact()¶
Context manager to perform a transaction by locking the index.
While the index is locked, no other write operation is permitted. Transactions should therefore be as short as possible. Read and write operations performed in a transaction are atomic. Read operations may occur concurrent to a transaction.
Transactions may be nested and may not be shared between threads.
>>> from diskcache import Index >>> mapping = Index() >>> with mapping.transact(): # Atomically increment two keys. ... mapping['total'] = mapping.get('total', 0) + 123.4 ... mapping['count'] = mapping.get('count', 0) + 1 >>> with mapping.transact(): # Atomically calculate average. ... average = mapping['total'] / mapping['count'] >>> average 123.4
- Returns
context manager for use in with statement
- values()¶
Set-like object providing a view of index values.
>>> index = Index() >>> index.update({'a': 1, 'b': 2, 'c': 3}) >>> values_view = index.values() >>> 2 in values_view True
- Returns
values view
Recipes¶
- class diskcache.Averager(cache, key, expire=None, tag=None)¶
Recipe for calculating a running average.
Sometimes known as “online statistics,” the running average maintains the total and count. The average can then be calculated at any time.
>>> import diskcache >>> cache = diskcache.FanoutCache() >>> ave = Averager(cache, 'latency') >>> ave.add(0.080) >>> ave.add(0.120) >>> ave.get() 0.1 >>> ave.add(0.160) >>> ave.pop() 0.12 >>> print(ave.get()) None
- add(value)¶
Add value to average.
- get()¶
Get current average or return None if count equals zero.
- pop()¶
Return current average and delete key.
- class diskcache.Lock(cache, key, expire=None, tag=None)¶
Recipe for cross-process and cross-thread lock.
>>> import diskcache >>> cache = diskcache.Cache() >>> lock = Lock(cache, 'report-123') >>> lock.acquire() >>> lock.release() >>> with lock: ... pass
- acquire()¶
Acquire lock using spin-lock algorithm.
- locked()¶
Return true if the lock is acquired.
- release()¶
Release lock by deleting key.
- class diskcache.RLock(cache, key, expire=None, tag=None)¶
Recipe for cross-process and cross-thread re-entrant lock.
>>> import diskcache >>> cache = diskcache.Cache() >>> rlock = RLock(cache, 'user-123') >>> rlock.acquire() >>> rlock.acquire() >>> rlock.release() >>> with rlock: ... pass >>> rlock.release() >>> rlock.release() Traceback (most recent call last): ... AssertionError: cannot release un-acquired lock
- acquire()¶
Acquire lock by incrementing count using spin-lock algorithm.
- release()¶
Release lock by decrementing count.
- class diskcache.BoundedSemaphore(cache, key, value=1, expire=None, tag=None)¶
Recipe for cross-process and cross-thread bounded semaphore.
>>> import diskcache >>> cache = diskcache.Cache() >>> semaphore = BoundedSemaphore(cache, 'max-cons', value=2) >>> semaphore.acquire() >>> semaphore.acquire() >>> semaphore.release() >>> with semaphore: ... pass >>> semaphore.release() >>> semaphore.release() Traceback (most recent call last): ... AssertionError: cannot release un-acquired semaphore
- acquire()¶
Acquire semaphore by decrementing value using spin-lock algorithm.
- release()¶
Release semaphore by incrementing value.
- @diskcache.throttle(cache, count, seconds, name=None, expire=None, tag=None, time_func=<built-in function time>, sleep_func=<built-in function sleep>)¶
Decorator to throttle calls to function.
>>> import diskcache, time >>> cache = diskcache.Cache() >>> count = 0 >>> @throttle(cache, 2, 1) # 2 calls per 1 second ... def increment(): ... global count ... count += 1 >>> start = time.time() >>> while (time.time() - start) <= 2: ... increment() >>> count in (6, 7) # 6 or 7 calls depending on CPU load True
- @diskcache.barrier(cache, lock_factory, name=None, expire=None, tag=None)¶
Barrier to calling decorated function.
Supports different kinds of locks: Lock, RLock, BoundedSemaphore.
>>> import diskcache, time >>> cache = diskcache.Cache() >>> @barrier(cache, Lock) ... def work(num): ... print('worker started') ... time.sleep(1) ... print('worker finished') >>> import multiprocessing.pool >>> pool = multiprocessing.pool.ThreadPool(2) >>> _ = pool.map(work, range(2)) worker started worker finished worker started worker finished >>> pool.terminate()
- @diskcache.memoize_stampede(cache, expire, name=None, typed=False, tag=None, beta=1, ignore=())¶
Memoizing cache decorator with cache stampede protection.
Cache stampedes are a type of system overload that can occur when parallel computing systems using memoization come under heavy load. This behaviour is sometimes also called dog-piling, cache miss storm, cache choking, or the thundering herd problem.
The memoization decorator implements cache stampede protection through early recomputation. Early recomputation of function results will occur probabilistically before expiration in a background thread of execution. Early probabilistic recomputation is based on research by Vattani, A.; Chierichetti, F.; Lowenstein, K. (2015), Optimal Probabilistic Cache Stampede Prevention, VLDB, pp. 886-897, ISSN 2150-8097
If name is set to None (default), the callable name will be determined automatically.
If typed is set to True, function arguments of different types will be cached separately. For example, f(3) and f(3.0) will be treated as distinct calls with distinct results.
The original underlying function is accessible through the __wrapped__ attribute. This is useful for introspection, for bypassing the cache, or for rewrapping the function with a different cache.
>>> from diskcache import Cache >>> cache = Cache() >>> @memoize_stampede(cache, expire=1) ... def fib(number): ... if number == 0: ... return 0 ... elif number == 1: ... return 1 ... else: ... return fib(number - 1) + fib(number - 2) >>> print(fib(100)) 354224848179261915075
An additional __cache_key__ attribute can be used to generate the cache key used for the given arguments.
>>> key = fib.__cache_key__(100) >>> del cache[key]
Remember to call memoize when decorating a callable. If you forget, then a TypeError will occur.
- Parameters
cache – cache to store callable arguments and return values
expire (float) – seconds until arguments expire
name (str) – name given for callable (default None, automatic)
typed (bool) – cache different types separately (default False)
tag (str) – text to associate with arguments (default None)
ignore (set) – positional or keyword args to ignore (default ())
- Returns
callable decorator
Constants¶
Read the Settings tutorial for details.
- diskcache.DEFAULT_SETTINGS¶
statistics (int) default 0 - disabled when 0, enabled when 1.
tag_index (int) default 0 - disabled when 0, enabled when 1.
eviction_policy (str) default “least-recently-stored” - any of the keys in EVICTION_POLICY as described below.
size_limit (int, in bytes) default one gigabyte - approximate size limit of cache.
cull_limit (int) default ten - maximum number of items culled during set or add operations.
sqlite_auto_vacuum (int) default 1, “FULL” - SQLite auto vacuum pragma.
sqlite_cache_size (int, in pages) default 8,192 - SQLite cache size pragma.
sqlite_journal_mode (str) default “wal” - SQLite journal mode pragma.
sqlite_mmap_size (int, in bytes) default 64 megabytes - SQLite mmap size pragma.
sqlite_synchronous (int) default 1, “NORMAL” - SQLite synchronous pragma.
disk_min_file_size (int, in bytes) default 32 kilobytes - values with greater size are stored in files.
disk_pickle_protocol (int) default highest Pickle protocol - the Pickle protocol to use for data types that are not natively supported.
- diskcache.EVICTION_POLICY¶
least-recently-stored (default) - evict least recently stored keys first.
least-recently-used - evict least recently used keys first.
least-frequently-used - evict least frequently used keys first.
none - never evict keys.
Disk¶
Read the Disk tutorial for details.
- class diskcache.Disk(directory, min_file_size=0, pickle_protocol=0)¶
Cache key and value serialization for SQLite database and files.
- __init__(directory, min_file_size=0, pickle_protocol=0)¶
Initialize disk instance.
- Parameters
directory (str) – directory path
min_file_size (int) – minimum size for file use
pickle_protocol (int) – pickle protocol for serialization
- fetch(mode, filename, value, read)¶
Convert fields mode, filename, and value from Cache table to value.
- Parameters
mode (int) – value mode raw, binary, text, or pickle
filename (str) – filename of corresponding value
value – database value
read (bool) – when True, return an open file handle
- Returns
corresponding Python value
- Raises
IOError if the value cannot be read
- filename(key=UNKNOWN, value=UNKNOWN)¶
Return filename and full-path tuple for file storage.
Filename will be a randomly generated 28 character hexadecimal string with “.val” suffixed. Two levels of sub-directories will be used to reduce the size of directories. On older filesystems, lookups in directories with many files may be slow.
The default implementation ignores the key and value parameters.
In some scenarios, for example
Cache.push
, the key or value may not be known when the item is stored in the cache.- Parameters
key – key for item (default UNKNOWN)
value – value for item (default UNKNOWN)
- get(key, raw)¶
Convert fields key and raw from Cache table to key.
- Parameters
key – database key to convert
raw (bool) – flag indicating raw database storage
- Returns
corresponding Python key
- hash(key)¶
Compute portable hash for key.
- Parameters
key – key to hash
- Returns
hash value
- put(key)¶
Convert key to fields key and raw for Cache table.
- Parameters
key – key to convert
- Returns
(database key, raw boolean) pair
- remove(file_path)¶
Remove a file given by file_path.
This method is cross-thread and cross-process safe. If an OSError occurs, it is suppressed.
- Parameters
file_path (str) – relative path to file
- store(value, read, key=UNKNOWN)¶
Convert value to fields size, mode, filename, and value for Cache table.
- Parameters
value – value to convert
read (bool) – True when value is file-like object
key – key for item (default UNKNOWN)
- Returns
(size, mode, filename, value) tuple for Cache table
JSONDisk¶
Read the Disk tutorial for details.
- class diskcache.JSONDisk(directory, compress_level=1, **kwargs)¶
Cache key and value using JSON serialization with zlib compression.
- __init__(directory, compress_level=1, **kwargs)¶
Initialize JSON disk instance.
Keys and values are compressed using the zlib library. The compress_level is an integer from 0 to 9 controlling the level of compression; 1 is fastest and produces the least compression, 9 is slowest and produces the most compression, and 0 is no compression.
- Parameters
directory (str) – directory path
compress_level (int) – zlib compression level (default 1)
kwargs – super class arguments
- fetch(mode, filename, value, read)¶
Convert fields mode, filename, and value from Cache table to value.
- Parameters
mode (int) – value mode raw, binary, text, or pickle
filename (str) – filename of corresponding value
value – database value
read (bool) – when True, return an open file handle
- Returns
corresponding Python value
- Raises
IOError if the value cannot be read
- get(key, raw)¶
Convert fields key and raw from Cache table to key.
- Parameters
key – database key to convert
raw (bool) – flag indicating raw database storage
- Returns
corresponding Python key
- put(key)¶
Convert key to fields key and raw for Cache table.
- Parameters
key – key to convert
- Returns
(database key, raw boolean) pair
- store(value, read, key=UNKNOWN)¶
Convert value to fields size, mode, filename, and value for Cache table.
- Parameters
value – value to convert
read (bool) – True when value is file-like object
key – key for item (default UNKNOWN)
- Returns
(size, mode, filename, value) tuple for Cache table
Timeout¶
- exception diskcache.Timeout¶
Database timeout expired.