devilry.apps.core.deliverystore
— DeliveryStore¶
Deprecated since version 3.0: This module is deprecated in 3.0 and will be removed in a future release.
A DeliveryStore is a place to put the files from deliveries. In more
technical terms, it is a place where each file related to a
devilry.apps.core.models.FileMeta
is stored.
Selecting a DeliveryStore¶
Devilry on comes with one DeliveryStore ready for production use,
FsDeliveryStore
. To enable a DeliveryStore, you have to set the
DELIVERY_STORE_BACKEND
-setting in your settings.py like this:
DELIVERY_STORE_BACKEND = 'devilry.apps.core.deliverystore.FsDeliveryStore'
The FsDeliveryStore also require you to define where on the disk you wish to
store your files in the DELIVERY_STORE_ROOT
-setting like this:
DELIVERY_STORE_ROOT = '/path/to/root/directory/of/my/deliverystore'
Creating your own DeliveryStore¶
To create your own DeliveryStore you have to implement
DeliveryStoreInterface
. A good example is FsDeliveryStore
:
class FsDeliveryStore(DeliveryStoreInterface): """ Filesystem-based DeliveryStore suitable for production use. It stores files in a filesystem hierarcy with one directory for each Delivery, with the delivery-id as name. In each delivery-directory, the files are stored by FileMeta id. """ def __init__(self, root=None): """ :param root: The root-directory where files are stored. Defaults to the value of the ``DELIVERY_STORE_ROOT``-setting. """ self.root = root or settings.DELIVERY_STORE_ROOT def _get_dirpath(self, delivery_obj): return join(self.root, str(delivery_obj.pk)) def _get_filepath(self, filemeta_obj): return join(self._get_dirpath(filemeta_obj.delivery), str(filemeta_obj.pk)) def read_open(self, filemeta_obj): filepath = self._get_filepath(filemeta_obj) if not exists(filepath): raise FileNotFoundError(filemeta_obj) return open(filepath, 'rb') def _create_dir(self, filemeta_obj): dirpath = self._get_dirpath(filemeta_obj.delivery) if not exists(dirpath): makedirs(dirpath) def write_open(self, filemeta_obj): self._create_dir(filemeta_obj) return open(self._get_filepath(filemeta_obj), 'wb') def remove(self, filemeta_obj): filepath = self._get_filepath(filemeta_obj) if not exists(filepath): raise FileNotFoundError(filemeta_obj) remove(filepath) def exists(self, filemeta_obj): filepath = self._get_filepath(filemeta_obj) return exists(filepath) def copy(self, filemeta_obj_from, filemeta_obj_to): frompath = self._get_filepath(filemeta_obj_from) topath = self._get_filepath(filemeta_obj_to) self._create_dir(filemeta_obj_to) shutil_copy(frompath, topath)
- class devilry.apps.core.testhelpers.DeliveryStoreTestMixin¶
Mixin-class that tests if
devilry.core.deliverystore.DeliveryStoreInterface
is implemented correctly.You only need to override
get_storageobj()
, and maybesetUp()
andtearDown()
, but make sure you callsuper(..., self).setUp()
if you override it.You must mixin this class before
django.test.TestCase
like so:class TestMyDeliveryStore(DeliveryStoreTestMixin, django.test.TestCase): ...
Setting the DeliveryStore manually - for tests¶
You might need to set the DeliveryStore manually if you need to handle
deliveries in your own tests. Just set devilry.apps.core.FileMeta.deliveryStore
like this:
from django.test import TestCase
from devilry.apps.core.models import FileMeta, Delivery
from devilry.apps.core.deliverystore import MemoryDeliveryStore
class MyTest(TestCase):
def test_something(self):
FileMeta.deliverystore = MemoryDeliveryStore()
delivery = Delivery.begin(assignmentgroup, user)
delivery.add_file('hello.txt', ['hello', 'world'])
delivery.finish()
The recommended production deliverystore¶
- If you store files in a traditional file system:
The recommended DeliveryStore for traditional filesystems is
devilry.apps.core.deliverystore.FsHierDeliveryStore
.It stores files in a filesystem hierarcy with one directory for each Delivery, with the delivery-id as name. In each delivery-directory, the files are stored by FileMeta id.
- If you store files in a modern cloud file system like Amazon S3:
The recommended DeliveryStore for cloud file systems that have no upper limit on the number of files in a single directory is
devilry.apps.core.deliverystore.DjangoStorageDeliveryStore
.
Directory hierachy¶
The delivery directories are stored in a
hierarchy with two parent directories. The parent directories are numeric intervals.
We have one top-level directory for each N in interval_size*interval_size*N
. Within each
toplevel directory, we have one subdirectory for each N in
interval_size*N
.
Directory hierarchy example¶
For interval_size
of 1000
, this will use the following hierarchy:
0/
0/
0/
1/
.
.
1/
1000/
2000/
.
.
2/
.
.
999/
1/
0/
1000000/
1000001/
.
.
1/
1001000/
1001001/
2/
.
.
999/
2/
.
.
999/
API¶
- exception devilry.apps.core.deliverystore.FileNotFoundError(filemeta_obj)¶
Exception to be raised when the remove method of a DeliveryStore does not find the given file.
- class devilry.apps.core.deliverystore.MemFile(initial_value='', newline='\n')¶
- close()¶
Close the IO object.
Attempting any further operation after the object is closed will raise a ValueError.
This method has no effect if the file is already closed.
- class devilry.apps.core.deliverystore.DeliveryStoreInterface¶
The interface all deliverystores must implement. All methods raise
NotImplementedError
.- read_open(filemeta_obj)¶
Return a file-like object opened for reading.
The returned object must have
close()
andread()
methods as defined by the documentation of the standard python file-class.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- write_open(filemeta_obj)¶
Return a file-like object opened for writing.
The returned object must have
close()
andwrite()
methods as defined by the documentation of the standard python file-class.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- remove(filemeta_obj)¶
Remove the file.
Note that this method is called before the filemeta_obj is removed. This means that the file might be removed, and the removal of the filemeta_obj can still fail. To prevent users from having to manually resolve such cases implementations should check if the file exists, and raise FileNotFoundError if it does not.
The calling function has to check for FileNotFoundError and handle any other error.
- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- exists(filemeta_obj)¶
Return
True
if the file exists,False
if not.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- copy(filemeta_obj_from, filemeta_obj_to)¶
Copy the underlying file-object for
filemeta_obj_from
into the file-object forfilemeta_obj_to
.Defaults to an inefficient implementation using
read_open()
and meth:.write_open. Should be overridden for backends with some form of native copy-capability.
- class devilry.apps.core.deliverystore.FsDeliveryStore(root=None)¶
Filesystem-based DeliveryStore suitable for production use.
It stores files in a filesystem hierarcy with one directory for each Delivery, with the delivery-id as name. In each delivery-directory, the files are stored by FileMeta id.
- Parameters:
root – The root-directory where files are stored. Defaults to the value of the
DELIVERY_STORE_ROOT
-setting.
- read_open(filemeta_obj)¶
Return a file-like object opened for reading.
The returned object must have
close()
andread()
methods as defined by the documentation of the standard python file-class.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- write_open(filemeta_obj)¶
Return a file-like object opened for writing.
The returned object must have
close()
andwrite()
methods as defined by the documentation of the standard python file-class.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- remove(filemeta_obj)¶
Remove the file.
Note that this method is called before the filemeta_obj is removed. This means that the file might be removed, and the removal of the filemeta_obj can still fail. To prevent users from having to manually resolve such cases implementations should check if the file exists, and raise FileNotFoundError if it does not.
The calling function has to check for FileNotFoundError and handle any other error.
- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- exists(filemeta_obj)¶
Return
True
if the file exists,False
if not.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- copy(filemeta_obj_from, filemeta_obj_to)¶
Copy the underlying file-object for
filemeta_obj_from
into the file-object forfilemeta_obj_to
.Defaults to an inefficient implementation using
read_open()
and meth:.write_open. Should be overridden for backends with some form of native copy-capability.
- class devilry.apps.core.deliverystore.FsHierDeliveryStore(root=None, interval=None)¶
Filesystem-based DeliveryStore suitable for production use with huge amounts of deliveries.
- Parameters:
root – The root-directory where files are stored. Defaults to the value of the
DEVILRY_FSHIERDELIVERYSTORE_ROOT
-setting.interval – The interval. Defaults to the value of the
DEVILRY_FSHIERDELIVERYSTORE_INTERVAL
-setting.
- get_path_from_deliveryid(deliveryid)¶
>>> fs = FsHierDeliveryStore('/stuff/', interval=1000) >>> fs.get_path_from_deliveryid(deliveryid=2001000) (2, 1) >>> fs.get_path_from_deliveryid(deliveryid=1000) (0, 1) >>> fs.get_path_from_deliveryid(deliveryid=1005) (0, 1) >>> fs.get_path_from_deliveryid(deliveryid=2005) (0, 2) >>> fs.get_path_from_deliveryid(deliveryid=0) (0, 0) >>> fs.get_path_from_deliveryid(deliveryid=1) (0, 0) >>> fs.get_path_from_deliveryid(deliveryid=1000000) (1, 0)
- class devilry.apps.core.deliverystore.MemoryDeliveryStore¶
Memory-base DeliveryStore ONLY FOR TESTING.
This is only for testing, and it does not handle parallel access. Suitable for unittesting.
- read_open(filemeta_obj)¶
Return a file-like object opened for reading.
The returned object must have
close()
andread()
methods as defined by the documentation of the standard python file-class.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- write_open(filemeta_obj)¶
Return a file-like object opened for writing.
The returned object must have
close()
andwrite()
methods as defined by the documentation of the standard python file-class.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- remove(filemeta_obj)¶
Remove the file.
Note that this method is called before the filemeta_obj is removed. This means that the file might be removed, and the removal of the filemeta_obj can still fail. To prevent users from having to manually resolve such cases implementations should check if the file exists, and raise FileNotFoundError if it does not.
The calling function has to check for FileNotFoundError and handle any other error.
- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- exists(filemeta_obj)¶
Return
True
if the file exists,False
if not.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- class devilry.apps.core.deliverystore.DjangoStorageDeliveryStore(root=None, storage_backend=None)¶
Delivery store backend that uses Django storages.
Initialize the deliverystore with an optional root directory (relative to MEDIA_ROOT) and optionally a custom storage backend.
- Parameters:
root – The root-directory relative to
MEDIA_ROOT
where files are stored. Defaults to the value of theDELIVERY_STORE_ROOT
-setting.storage_backend – The django storage backend to use. Defaults to
django.core.files.storage.default_storage
.
- read_open(filemeta_obj)¶
Return a file-like object opened for reading.
The returned object must have
close()
andread()
methods as defined by the documentation of the standard python file-class.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- write_open(filemeta_obj)¶
Return a file-like object opened for writing.
The returned object must have
close()
andwrite()
methods as defined by the documentation of the standard python file-class.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- remove(filemeta_obj)¶
Remove the file.
Note that this method is called before the filemeta_obj is removed. This means that the file might be removed, and the removal of the filemeta_obj can still fail. To prevent users from having to manually resolve such cases implementations should check if the file exists, and raise FileNotFoundError if it does not.
The calling function has to check for FileNotFoundError and handle any other error.
- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- exists(filemeta_obj)¶
Return
True
if the file exists,False
if not.- Parameters:
filemeta_obj – A
devilry.core.models.FileMeta
-object.
- copy(filemeta_obj_from, filemeta_obj_to)¶
Copy the underlying file-object for
filemeta_obj_from
into the file-object forfilemeta_obj_to
.Defaults to an inefficient implementation using
read_open()
and meth:.write_open. Should be overridden for backends with some form of native copy-capability.