devilry_compressionutil
— Devilry utils models and backends¶
The devilry_compressionutil
handles the creation of compressed archives.
About the devilry compressionutil app¶
The devilry_compressionutil
app provides utilities for creating and adding archives using
ZipUtil
, by providing a backend and a registry for the specialized backends to use.
devilry_compressionutil
also provides a meta class used for caching metainfo about a archive.
How to define a backend¶
Add path setting
First off, add a path to the storage location Django settings config.
Call this setting DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY
DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY = os.path.join('path', 'to', 'storage', 'location')
Subclass backend
Create a backend for your app that you will use to compress the files by subclassing e.g
PythonZipFileBackend
in devilry.devilry_compressionutil.backends.backends_base
.
from devilry.devilry_compressionutil.backends import backends_base
class YourAppZipBackend(backends_base.PythonZipFileBackend):
backend_id = 'some_backend_id'
def __init__(self, **kwargs):
super(YourAppZipBackend, self).__init__(**kwargs)
Load backend to registry
Create an AppConfig for your app.
Load the backend you created to the devilry_compressionutil
s registry.
Here you can register as many backends as you want, and just fetch it by its backend_id
.
The registry is a singleton, and to add a backend to it Registry.get_instance().add(..)
must be called.
...
from devilry.devilry_compressionutil import backend_registry
class YourAppAppConfig(AppConfig):
...
def ready(self):
from devilry.your_app import your_backends
backend_registry.Registry.get_instance().add(your_backends.YourAppZipBackend)
Get backend and add folders and files to it
Add paths and files to the backend.
from devilry.devilry_compressionutil import backend_registry
class HandleZipping:
...
def your_function_that_uses_zip_backend(...):
backend_class = backend_registry.Registry.get_instance().get('some_backend_id')
# Path to archive inside the location specified by the storage location defined in
# settings. ``DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY``
archive_name = 'archive_name'
full_archive_path = os.path.join('path', 'to', 'archive', 'archive_name')
# Instance the backend.
backend_instance = backend_class(
archive_path=full_archive_path,
archive_name=archive_name,
readmode=False
)
# Get files to compress
file1 = open('examplefile1.txt', 'r')
file2 = open('examplefile2.txt', 'r')
# Add file1 inside a folder named folder1
# Add file2 inside a folder named folder2
# The archive with content will now look like this
# archive_name.zip
# - folder1
# - examplefile1.txt
# - folder2
# - examplefile2.txt
backend_instance.add_file(os.path.join('folder1', 'examplefile1.txt'), file1)
backend_instance.add_file(os.path.join('folder2', 'examplefile2.txt'), file2)
# Close the backend
backend_instance.close()
# Read the archive as a fileobject
# ``read_archive()`` returns a FileObject of the archive
# This can for instance be passed to a HttpRequest.
backend_instance.readmode = True
fileobj = backend_instance.read_archive()
Datamodel API for caching metainfo about a archive¶
- class devilry.devilry_compressionutil.models.GenericMeta(*args, **kwargs)¶
Abstract class that implements usage of GenericForeignKey.
- content_type¶
Foreignkey to Djangos ContentType.
- content_object_id¶
ID of model to store.
- content_object¶
An arbitrary model to store.
- class devilry.devilry_compressionutil.models.CompressedArchiveMetaQueryset(model=None, query=None, using=None, hints=None)¶
Manager for class
CompressedArchiveMeta
.- create_meta(instance, zipfile_backend, user, user_role='')¶
Manager provides a way to create a meta entry for a archive. See
CompressedArchiveMeta
.- Parameters:
instance – Instance the archive is for.
zipfile_backend – base backend for compression, see
PythonZipFileBackend
.
- delete_compressed_archives_older_than(days=None, seconds=None)¶
Delete compressed archive meta entries older than a specified number of days or seconds.
- Parameters:
older_than_days (int) – Delete everything older than this.
older_than_seconds (int) – Delete everything older than this.
- delete_compressed_archives_marked_as_deleted()¶
Delete all compressed archives marked as deleted.
- class devilry.devilry_compressionutil.models.CompressedArchiveMeta(*args, **kwargs)¶
Metadata about a compressed archive. Name of the archive, path to it and it’s size.
- created_by¶
Who created the archive.
- CREATED_BY_ROLE_STUDENT = 'student'¶
Use this as value for
user_role
if the user is commenting as a student.
- CREATED_BY_ROLE_EXAMINER = 'examiner'¶
Use this as value for
user_role
if the user is commenting as an examiner.
- CREATED_BY_ROLE_ADMIN = 'admin'¶
Use this as value for
user_role
if the user is commenting as an admin.
- CREATED_BY_ROLE_CHOICES = (('student', 'Student'), ('examiner', 'Examiner'), ('admin', 'Admin'))¶
Choices for the
user_role
field.
- created_by_role¶
What role did the user create the archive with?
- created_datetime¶
When the archive was created.
- archive_name¶
SomeArchive2000.zip
.- Type:
The actual name of the archive, Example.
- archive_path¶
Path at storage location of the compressed archive. Example:
https://s3-eu-central-1.amazonaws.com/BUCKET/path/to/archive/SomeArchive2000.zip
- archive_size¶
Size of the archive in bytes.
- backend_id¶
The ID of the backend used. This is the ID attribute
backend_id
.
- deleted_datetime¶
When the entry was marked for deletion.
- clean()¶
Hook for doing any extra model-wide validation after clean() has been called on every field by self.clean_fields. Any ValidationError raised by this method will not be associated with a particular field; it will have a special-case association with the field defined by NON_FIELD_ERRORS.
- exception DoesNotExist¶
- exception MultipleObjectsReturned¶
- content_object¶
An arbitrary model to store.
- content_object_id¶
ID of model to store.
- content_type¶
Foreignkey to Djangos ContentType.
Backend base classes¶
- class devilry.devilry_compressionutil.backends.backends_base.BaseArchiveBackend(archive_path, archive_name='', readmode=True)¶
Specifies the interface for a backend compression-subclass.
All backends must implement this class.
- Parameters:
archive_path – Full path to archive including the archive name.
archive_name – Only the archive name.
readmode – Can be read from, defaults to
True
.
- backend_id = None¶
A unique string ID for the subclasses to use that describes what kind of backend it is.
- get_storage_location()¶
Get the storage location for archives.
- Returns:
Location specified in settings
DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY
.- Return type:
str
- read_binary()¶
Opens archive in read binary mode. Best suited for non-text files like images, videos etc or when serving file for download.
- Returns:
file object in read binary mode.
- Return type:
file
- Raises:
ValueError – If archive is
None
, orreadmode
is False.
- close()¶
Close archive when done with adding files to it.
- Raises:
ValueError – If
archive
isNone
.
- archive_size()¶
Get size of archive. Uses
os.stat
.- Returns:
size of archive.
- Return type:
int
- Raises:
ValueError – If not in
readmode
orarchive
isNone
.
- add_file(path, filelike_obj)¶
Add file to archive.
- Parameters:
path (str) – Path to the file inside the archive.
filelike_obj – An object which implements function
read()
.
- Raises:
NotImplementedError – If not implemented by subclass.
- read_archive()¶
Should return a object of the underlying compression tool in readmode.
- Raises:
NotImplementedError – If not implemented by subclass.
- get_archive()¶
Get the archive for compressed files.
- Raises:
NotImplementedError – If not implemented by subclass
- classmethod delete_archive(full_path)¶
Deletes the archive.
This must be implemented in subclass to handle the specifics for different storage backends.
- Parameters:
full_path (str) – Full path to the stored archive.
- Returns:
True
if deleted, elseFalse
.- Return type:
(boolean)
- class devilry.devilry_compressionutil.backends.backends_base.PythonZipFileBackend(**kwargs)¶
Defines a baseclass backend using
ZipFile
for theRegistry
.This class should be subclassed by backend-specific classes(backends for Heroku, S3, etc).
- Parameters:
archive_path – Full path to archive including the archive name.
archive_name – Only the archive name.
readmode – Can be read from, defaults to
True
.
- add_file(path, filelike_obj)¶
Add files to archive.
- Parameters:
path (str) – Path to the file inside the Zip-archive.
filelike_obj – An object with method
read()
- Raises:
ValueError – If
readmode
is set toTrue
, must beFalse
to add files.
- read_archive()¶
Get the zipped archive as
ZipFile
in readmode.- Returns:
The zipped archive.
- Return type:
ZipFile
- class devilry.devilry_compressionutil.backends.backends_base.PythonTarFileBackend(stream=False, compression='', **kwargs)¶
A baseclass backend using
TarFile
for archiving files to at tarball. Supports no compression,gzip
and``bzip2`` compression through the :class:’~TarFile’ class.- Parameters:
stream – If it should be handled as a stream or not.
compression – The compression mode used, defaults to
uncompressed
, but modesgz
,bz2
andxz
can also be used.
- compression_formats = ['', 'gz', 'bz2']¶
Compression formats supported.
- add_file(path, filelike_obj)¶
Writes a file to the archive on the given
path
.In fact, this creates a temporary folder hierarchy which is zipped when
closed()
is invoked on a instance of this class.- Parameters:
path – Path to file inside the archive.
filelike_obj – An object that behaves like a File(read, write..).
- read_archive()¶
Get TarFile in readmode.
- Returns:
The compressed tar archive.
- Return type:
TarFile
- close()¶
Must be invoked in order to finalize the archive by compressing the ‘temp’ directory to whichever compression level specified.
The temporary directory is then removed, and the tar archive is closed.
- get_archive()¶
Get TarFile(archive) in read-mode.
- Returns:
Opened in read mode.
- Return type:
TarFile