devilry.apps.core.deliverystore — DeliveryStore¶
A DeliveryStore is a place to put the files from deliveries. In more technical terms, it is a place where each file related to a devilry.apps.core.models.FileMeta is stored.
Selecting a DeliveryStore¶
Devilry on comes with one DeliveryStore ready for production use, FsDeliveryStore. To enable a DeliveryStore, you have to set the DELIVERY_STORE_BACKEND-setting in your settings.py like this:
DELIVERY_STORE_BACKEND = 'devilry.apps.core.deliverystore.FsDeliveryStore'
The FsDeliveryStore also require you to define where on the disk you wish to store your files in the DELIVERY_STORE_ROOT-setting like this:
DELIVERY_STORE_ROOT = '/path/to/root/directory/of/my/deliverystore'
Creating your own DeliveryStore¶
To create your own DeliveryStore you have to implement DeliveryStoreInterface. A good example is FsDeliveryStore:
class FsDeliveryStore(DeliveryStoreInterface): """ Filesystem-based DeliveryStore suitable for production use. It stores files in a filesystem hierarcy with one directory for each Delivery, with the delivery-id as name. In each delivery-directory, the files are stored by FileMeta id. """ def __init__(self, root=None): """ :param root: The root-directory where files are stored. Defaults to the value of the ``DELIVERY_STORE_ROOT``-setting. """ self.root = root or settings.DELIVERY_STORE_ROOT def _get_dirpath(self, delivery_obj): return join(self.root, str(delivery_obj.pk)) def _get_filepath(self, filemeta_obj): return join(self._get_dirpath(filemeta_obj.delivery), str(filemeta_obj.pk)) def read_open(self, filemeta_obj): filepath = self._get_filepath(filemeta_obj) if not exists(filepath): raise FileNotFoundError(filemeta_obj) return open(filepath, 'rb') def _create_dir(self, filemeta_obj): dirpath = self._get_dirpath(filemeta_obj.delivery) if not exists(dirpath): makedirs(dirpath) def write_open(self, filemeta_obj): self._create_dir(filemeta_obj) return open(self._get_filepath(filemeta_obj), 'wb') def remove(self, filemeta_obj): filepath = self._get_filepath(filemeta_obj) if not exists(filepath): raise FileNotFoundError(filemeta_obj) remove(filepath) def exists(self, filemeta_obj): filepath = self._get_filepath(filemeta_obj) return exists(filepath) def copy(self, filemeta_obj_from, filemeta_obj_to): frompath = self._get_filepath(filemeta_obj_from) topath = self._get_filepath(filemeta_obj_to) self._create_dir(filemeta_obj_to) shutil_copy(frompath, topath)
Testing your own DeliveryStore¶
We provide a mixing-class, devilry.apps.core.testhelpers.DeliveryStoreTestMixin, for you to extend when writing unit-tests for your DeliveryStore. Here is how we test FsDeliveryStore:
class TestFsDeliveryStore(DeliveryStoreTestMixin, TestCase): def setUp(self): self.root = mkdtemp() super(TestFsDeliveryStore, self).setUp() def get_storageobj(self): return FsDeliveryStore(self.root) def tearDown(self): rmtree(self.root)
- class devilry.apps.core.testhelpers.DeliveryStoreTestMixin¶
Bases: devilry.apps.core.testhelper.TestHelper
Mixin-class that tests if devilry.core.deliverystore.DeliveryStoreInterface is implemented correctly.
You only need to override get_storageobj(), and maybe setUp() and tearDown(), but make sure you call super(..., self).setUp() if you override it.
You must mixin this class before django.test.TestCase like so:
class TestMyDeliveryStore(DeliveryStoreTestMixin, django.test.TestCase): ...
- get_storageobj()¶
Return a object implementing devilry.core.deliverystore.DeliveryStoreInterface
- setUp()¶
Make sure to call this if you override it in subclasses, or the tests will fail.
Setting the DeliveryStore manually - for tests¶
You might need to set the DeliveryStore manually if you need to handle deliveries in your own tests. Just set devilry.apps.core.FileMeta.deliveryStore like this:
from django.test import TestCase
from devilry.apps.core.models import FileMeta, Delivery
from devilry.apps.core.deliverystore import MemoryDeliveryStore
class MyTest(TestCase):
def test_something(self):
FileMeta.deliverystore = MemoryDeliveryStore()
delivery = Delivery.begin(assignmentgroup, user)
delivery.add_file('hello.txt', ['hello', 'world'])
delivery.finish()
The recommended production deliverystore¶
The recommended DeliveryStore is devilry.apps.core.deliverystore.FsHierDeliveryStore.
It stores files in a filesystem hierarcy with one directory for each Delivery, with the delivery-id as name. In each delivery-directory, the files are stored by FileMeta id.
Directory hierachy¶
The delivery directories are stored in a hierarchy with two parent directories. The parent directories are numeric intervals. We have one top-level directory for each N in interval_size*interval_size*N. Within each toplevel directory, we have one subdirectory for each N in interval_size*N.
Directory hierarchy example¶
For interval_size of 1000, this will use the following hierarchy:
0/
0/
0/
1/
.
.
1/
1000/
2000/
.
.
2/
.
.
999/
1/
0/
1000000/
1000001/
.
.
1/
1001000/
1001001/
2/
.
.
999/
2/
.
.
999/
API¶
- exception devilry.apps.core.deliverystore.FileNotFoundError(filemeta_obj)¶
Bases: exceptions.Exception
Exception to be raised when the remove method of a DeliveryStore does not find the given file.
- class devilry.apps.core.deliverystore.DeliveryStoreInterface¶
Bases: object
The interface all deliverystores must implement. All methods raise NotImplementedError.
- read_open(filemeta_obj)¶
Return a file-like object opened for reading.
The returned object must have close() and read() methods as defined by the documentation of the standard python file-class.
Parameters: filemeta_obj – A devilry.core.models.FileMeta-object.
- write_open(filemeta_obj)¶
Return a file-like object opened for writing.
The returned object must have close() and write() methods as defined by the documentation of the standard python file-class.
Parameters: filemeta_obj – A devilry.core.models.FileMeta-object.
- remove(filemeta_obj)¶
Remove the file.
Note that this method is called before the filemeta_obj is removed. This means that the file might be removed, and the removal of the filemeta_obj can still fail. To prevent users from having to manually resolve such cases implementations should check if the file exists, and raise FileNotFoundError if it does not.
The calling function has to check for FileNotFoundError and handle any other error.
Parameters: filemeta_obj – A devilry.core.models.FileMeta-object.
- exists(filemeta_obj)¶
Return True if the file exists, False if not.
Parameters: filemeta_obj – A devilry.core.models.FileMeta-object.
- copy(filemeta_obj_from, filemeta_obj_to)¶
Copy the underlying file-object for filemeta_obj_from into the file-object for filemeta_obj_to.
Defaults to an inefficient implementation using read_open() and meth:.write_open. Should be overridden for backends with some form of native copy-capability.
- class devilry.apps.core.deliverystore.FsDeliveryStore(root=None)¶
Bases: devilry.apps.core.deliverystore.DeliveryStoreInterface
Filesystem-based DeliveryStore suitable for production use.
It stores files in a filesystem hierarcy with one directory for each Delivery, with the delivery-id as name. In each delivery-directory, the files are stored by FileMeta id.
Parameters: root – The root-directory where files are stored. Defaults to the value of the DELIVERY_STORE_ROOT-setting.
- class devilry.apps.core.deliverystore.FsHierDeliveryStore(root=None, interval=None)¶
Bases: devilry.apps.core.deliverystore.FsDeliveryStore
Filesystem-based DeliveryStore suitable for production use with huge amounts of deliveries.
Parameters: - root – The root-directory where files are stored. Defaults to the value of the DEVILRY_FSHIERDELIVERYSTORE_ROOT-setting.
- interval – The interval. Defaults to the value of the DEVILRY_FSHIERDELIVERYSTORE_INTERVAL-setting.
- get_path_from_deliveryid(deliveryid)¶
>>> fs = FsHierDeliveryStore('/stuff/', interval=1000) >>> fs.get_path_from_deliveryid(deliveryid=2001000) (2, 1) >>> fs.get_path_from_deliveryid(deliveryid=1000) (0, 1) >>> fs.get_path_from_deliveryid(deliveryid=1005) (0, 1) >>> fs.get_path_from_deliveryid(deliveryid=2005) (0, 2) >>> fs.get_path_from_deliveryid(deliveryid=0) (0, 0) >>> fs.get_path_from_deliveryid(deliveryid=1) (0, 0) >>> fs.get_path_from_deliveryid(deliveryid=1000000) (1, 0)
- class devilry.apps.core.deliverystore.MemoryDeliveryStore¶
Bases: devilry.apps.core.deliverystore.DeliveryStoreInterface
Memory-base DeliveryStore ONLY FOR TESTING.
This is only for testing, and it does not handle parallel access. Suitable for unittesting.