Storage¶
Backends¶
PostgreSQL¶
-
class
kinto.core.storage.postgresql.
Storage
(client, max_fetch_size, *args, **kwargs)¶ Storage backend using PostgreSQL.
Recommended in production (requires PostgreSQL 9.4 or higher).
Enable in configuration:
kinto.storage_backend = kinto.core.storage.postgresql
Database location URI can be customized:
kinto.storage_url = postgres://user:pass@db.server.lan:5432/dbname
Alternatively, username and password could also rely on system user ident or even specified in
~/.pgpass
(see PostgreSQL documentation).Note
Some tables and indices are created when
kinto migrate
is run. This requires some privileges on the database, or some error will be raised.Alternatively, the schema can be initialized outside the python application, using the SQL file located in
kinto/core/storage/postgresql/schema.sql
. This allows to distinguish schema manipulation privileges from schema usage.A connection pool is enabled by default:
kinto.storage_pool_size = 10 kinto.storage_maxoverflow = 10 kinto.storage_max_backlog = -1 kinto.storage_pool_recycle = -1 kinto.storage_pool_timeout = 30 kinto.cache_poolclass = kinto.core.storage.postgresql.pool.QueuePoolWithMaxBacklog
The
max_backlog
limits the number of threads that can be in the queue waiting for a connection. Once this limit has been reached, any further attempts to acquire a connection will be rejected immediately, instead of locking up all threads by keeping them waiting in the queue.See dedicated section in SQLAlchemy documentation for default values and behaviour.
Note
Using a dedicated connection pool is still recommended to allow load balancing, replication or limit the number of connections used in a multi-process deployment.
Redis¶
-
class
kinto.core.storage.redis.
Storage
(client, *args, **kwargs)¶ Storage backend implementation using Redis.
Warning
Useful for very low server load, but won’t scale since records sorting and filtering are performed in memory.
Enable in configuration:
kinto.storage_backend = kinto.core.storage.redis
(Optional) Instance location URI can be customized:
kinto.storage_url = redis://localhost:6379/0
A threaded connection pool is enabled by default:
kinto.storage_pool_size = 50
API¶
Implementing a custom storage backend consists in implementating the following interface:
-
class
kinto.core.storage.
Filter
(field, value, operator)¶ Filtering properties.
-
field
¶ Alias for field number 0
-
operator
¶ Alias for field number 2
-
value
¶ Alias for field number 1
-
-
class
kinto.core.storage.
Sort
(field, direction)¶ Sorting properties.
-
direction
¶ Alias for field number 1
-
field
¶ Alias for field number 0
-
-
class
kinto.core.storage.
StorageBase
¶ Storage abstraction used by resource views.
It is meant to be instantiated at application startup. Any operation may raise a HTTPServiceUnavailable error if an error occurs with the underlying service.
Configuration can be changed to choose which storage backend will persist the objects.
Raises: HTTPServiceUnavailable
-
initialize_schema
()¶ Create every necessary objects (like tables or indices) in the backend.
This is executed when the
kinto migrate
command is run.
-
flush
(auth=None)¶ Remove every object from this storage.
-
collection_timestamp
(collection_id, parent_id, auth=None)¶ Get the highest timestamp of every objects in this collection_id for this parent_id.
Note
This should take deleted objects into account.
Parameters: - collection_id (str) – the collection id.
- parent_id (str) – the collection parent.
Returns: the latest timestamp of the collection.
Return type: int
-
create
(collection_id, parent_id, object, id_generator=None, unique_fields=None, id_field='id', modified_field='last_modified', auth=None)¶ Create the specified object in this collection_id for this parent_id. Assign the id to the object, using the attribute
kinto.core.resource.Model.id_field
.Note
This will update the collection timestamp.
Raises: Parameters: - collection_id (str) – the collection id.
- parent_id (str) – the collection parent.
- object (dict) – the object to create.
Returns: the newly created object.
Return type: dict
-
get
(collection_id, parent_id, object_id, id_field='id', modified_field='last_modified', auth=None)¶ Retrieve the object with specified object_id, or raise error if not found.
Raises: Parameters: - collection_id (str) – the collection id.
- parent_id (str) – the collection parent.
- object_id (str) – unique identifier of the object
Returns: the object object.
Return type: dict
-
update
(collection_id, parent_id, object_id, object, unique_fields=None, id_field='id', modified_field='last_modified', auth=None)¶ Overwrite the object with the specified object_id.
If the specified id is not found, the object is created with the specified id.
Note
This will update the collection timestamp.
Raises: Parameters: - collection_id (str) – the collection id.
- parent_id (str) – the collection parent.
- object_id (str) – unique identifier of the object
- object (dict) – the object to update or create.
Returns: the updated object.
Return type: dict
-
delete
(collection_id, parent_id, object_id, with_deleted=True, id_field='id', modified_field='last_modified', deleted_field='deleted', auth=None)¶ Delete the object with specified object_id, and raise error if not found.
Deleted objects must be removed from the database, but their ids and timestamps of deletion must be tracked for synchronization purposes. (See
kinto.core.storage.StorageBase.get_all()
)Note
This will update the collection timestamp.
Raises: Parameters: - collection_id (str) – the collection id.
- parent_id (str) – the collection parent.
- object_id (str) – unique identifier of the object
- with_deleted (bool) – track deleted record with a tombstone
Returns: the deleted object, with minimal set of attributes.
Return type: dict
-
delete_all
(collection_id, parent_id, filters=None, with_deleted=True, id_field='id', modified_field='last_modified', deleted_field='deleted', auth=None)¶ Delete all objects in this collection_id for this parent_id.
Parameters: - collection_id (str) – the collection id.
- parent_id (str) – the collection parent.
- filters (list of
kinto.core.storage.Filter
) – Optionnally filter the objects to delete. - with_deleted (bool) – track deleted records with a tombstone
Returns: the list of deleted objects, with minimal set of attributes.
Return type: list of dict
-
purge_deleted
(collection_id, parent_id, before=None, id_field='id', modified_field='last_modified', auth=None)¶ Delete all deleted object tombstones in this collection_id for this parent_id.
Parameters: - collection_id (str) – the collection id.
- parent_id (str) – the collection parent.
- before (int) – Optionnal timestamp to limit deletion (exclusive)
Returns: The number of deleted objects.
Return type: int
-
get_all
(collection_id, parent_id, filters=None, sorting=None, pagination_rules=None, limit=None, include_deleted=False, id_field='id', modified_field='last_modified', deleted_field='deleted', auth=None)¶ Retrieve all objects in this collection_id for this parent_id.
Parameters: - collection_id (str) – the collection id.
- parent_id (str) – the collection parent.
- filters (list of
kinto.core.storage.Filter
) – Optionally filter the objects by their attribute. Each filter in this list is a tuple of a field, a value and a comparison (see kinto.core.utils.COMPARISON). All filters are combined using AND. - sorting (list of
kinto.core.storage.Sort
) – Optionnally sort the objects by attribute. Each sort instruction in this list refers to a field and a direction (negative means descending). All sort instructions are cumulative. - pagination_rules (list of list of
kinto.core.storage.Filter
) – Optionnally paginate the list of objects. This list of rules aims to reduce the set of objects to the current page. A rule is a list of filters (see filters parameter), and all rules are combined using OR. - limit (int) – Optionnally limit the number of objects to be retrieved.
- include_deleted (bool) – Optionnally include the deleted objects that match the filters.
Returns: the limited list of objects, and the total number of matching objects in the collection (deleted ones excluded).
Return type: tuple (list, integer)
-
Exceptions¶
Exceptions raised by storage backend.
-
exception
kinto.core.storage.exceptions.
BackendError
(original=None, message=None, *args, **kwargs)¶ A generic exception raised by storage on error.
Parameters: original (Exception) – the wrapped exception raised by underlying library.
-
exception
kinto.core.storage.exceptions.
RecordNotFoundError
¶ An exception raised when a specific record could not be found.
-
exception
kinto.core.storage.exceptions.
UnicityError
(field, record, *args, **kwargs)¶ An exception raised on unicity constraint violation.
Raised by storage backend when the creation or the modification of a record violates the unicity constraints defined by the resource.
Store custom data¶
Storage can be used to store arbitrary data.
data = {'subscribed': datetime.now()}
user_id = request.authenticated_userid
storage = request.registry.storage
storage.create(collection_id='__custom', parent_id='', record=data)
See the Model class to manipulate collections of records.