`devilry_compressionutil` — Devilry utils models and backends

The devilry_compressionutil handles the creation of compressed archives.

About the devilry compressionutil app

The devilry_compressionutil app provides utilities for creating and adding archives using ZipUtil, by providing a backend and a registry for the specialized backends to use.

devilry_compressionutil also provides a meta class used for caching metainfo about a archive.

How to define a backend

Add path setting

First off, add a path to the storage location Django settings config. Call this setting DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY

DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY = os.path.join('path', 'to', 'storage', 'location')

Subclass backend

Create a backend for your app that you will use to compress the files by subclassing e.g PythonZipFileBackend in devilry.devilry_compressionutil.backends.backends_base.

from devilry.devilry_compressionutil.backends import backends_base

class YourAppZipBackend(backends_base.PythonZipFileBackend):
    backend_id = 'some_backend_id'

    def __init__(self, **kwargs):
        super(YourAppZipBackend, self).__init__(**kwargs)

Load backend to registry

Create an AppConfig for your app. Load the backend you created to the devilry_compressionutil s registry. Here you can register as many backends as you want, and just fetch it by its backend_id. The registry is a singleton, and to add a backend to it Registry.get_instance().add(..) must be called.

...
from devilry.devilry_compressionutil import backend_registry

class YourAppAppConfig(AppConfig):
    ...

    def ready(self):
        from devilry.your_app import your_backends
        backend_registry.Registry.get_instance().add(your_backends.YourAppZipBackend)

Get backend and add folders and files to it

Add paths and files to the backend.

from devilry.devilry_compressionutil import backend_registry

class HandleZipping:
    ...

    def your_function_that_uses_zip_backend(...):
        backend_class = backend_registry.Registry.get_instance().get('some_backend_id')

        # Path to archive inside the location specified by the storage location defined in
        # settings. ``DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY``
        archive_name = 'archive_name'
        full_archive_path = os.path.join('path', 'to', 'archive', 'archive_name')

        # Instance the backend.
        backend_instance = backend_class(
            archive_path=full_archive_path,
            archive_name=archive_name,
            readmode=False
         )

        # Get files to compress
        file1 = open('examplefile1.txt', 'r')
        file2 = open('examplefile2.txt', 'r')

        # Add file1 inside a folder named folder1
        # Add file2 inside a folder named folder2
        # The archive with content will now look like this
        # archive_name.zip
        #     - folder1
        #         - examplefile1.txt
        #     - folder2
        #         - examplefile2.txt
        backend_instance.add_file(os.path.join('folder1', 'examplefile1.txt'), file1)
        backend_instance.add_file(os.path.join('folder2', 'examplefile2.txt'), file2)

        # Close the backend
        backend_instance.close()

        # Read the archive as a fileobject
        # ``read_archive()`` returns a FileObject of the archive
        # This can for instance be passed to a HttpRequest.
        backend_instance.readmode = True
        fileobj = backend_instance.read_archive()

Datamodel API for caching metainfo about a archive

class devilry.devilry_compressionutil.models.GenericMeta(*args, **kwargs)

Abstract class that implements usage of GenericForeignKey.

content_type: Foreignkey to Djangos ContentType.

content_object_id: ID of model to store.

content_object: An arbitrary model to store.

class devilry.devilry_compressionutil.models.CompressedArchiveMetaQueryset(model=None, query=None, using=None, hints=None)

Manager for class CompressedArchiveMeta.

create_meta(instance, zipfile_backend, user, user_role='')

Manager provides a way to create a meta entry for a archive. See CompressedArchiveMeta.

Parameters:

instance – Instance the archive is for.
zipfile_backend – base backend for compression, see PythonZipFileBackend.

delete_compressed_archives_older_than(days=None, seconds=None)

Delete compressed archive meta entries older than a specified number of days or seconds.

Parameters:

older_than_days (int) – Delete everything older than this.
older_than_seconds (int) – Delete everything older than this.

delete_compressed_archives_marked_as_deleted(): Delete all compressed archives marked as deleted.

class devilry.devilry_compressionutil.models.CompressedArchiveMeta(*args, **kwargs)

Metadata about a compressed archive. Name of the archive, path to it and it’s size.

created_by: Who created the archive.

CREATED_BY_ROLE_STUDENT = 'student': Use this as value for user_role if the user is commenting as a student.

CREATED_BY_ROLE_EXAMINER = 'examiner': Use this as value for user_role if the user is commenting as an examiner.

CREATED_BY_ROLE_ADMIN = 'admin': Use this as value for user_role if the user is commenting as an admin.

CREATED_BY_ROLE_CHOICES = (('student', 'Student'), ('examiner', 'Examiner'), ('admin', 'Admin')): Choices for the user_role field.

created_by_role: What role did the user create the archive with?

created_datetime: When the archive was created.

archive_name

SomeArchive2000.zip.

Type:: The actual name of the archive, Example.

archive_path: Path at storage location of the compressed archive. Example: https://s3-eu-central-1.amazonaws.com/BUCKET/path/to/archive/SomeArchive2000.zip

archive_size: Size of the archive in bytes.

backend_id: The ID of the backend used. This is the ID attribute backend_id.

deleted_datetime: When the entry was marked for deletion.

clean(): Hook for doing any extra model-wide validation after clean() has been called on every field by self.clean_fields. Any ValidationError raised by this method will not be associated with a particular field; it will have a special-case association with the field defined by NON_FIELD_ERRORS.

exception DoesNotExist

exception MultipleObjectsReturned

content_object: An arbitrary model to store.

content_object_id: ID of model to store.

content_type: Foreignkey to Djangos ContentType.

Backend base classes

class devilry.devilry_compressionutil.backends.backends_base.BaseArchiveBackend(archive_path, archive_name='', readmode=True)

Specifies the interface for a backend compression-subclass.

All backends must implement this class.

Parameters:

archive_path – Full path to archive including the archive name.
archive_name – Only the archive name.
readmode – Can be read from, defaults to True.

backend_id = None: A unique string ID for the subclasses to use that describes what kind of backend it is.

get_storage_location()

Get the storage location for archives.

Returns:: Location specified in settings DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY.
Return type:: str

read_binary()

Opens archive in read binary mode. Best suited for non-text files like images, videos etc or when serving file for download.

Returns:: file object in read binary mode.
Return type:: file
Raises:: ValueError – If archive is None, or readmode is False.

close()

Close archive when done with adding files to it.

Raises:: ValueError – If archive is None.

archive_size()

Get size of archive. Uses os.stat.

Returns:: size of archive.
Return type:: int
Raises:: ValueError – If not in readmode or archive is None.

add_file(path, filelike_obj)

Add file to archive.

Parameters:

path (str) – Path to the file inside the archive.
filelike_obj – An object which implements function read().

Raises:

NotImplementedError – If not implemented by subclass.

read_archive()

Should return a object of the underlying compression tool in readmode.

Raises:: NotImplementedError – If not implemented by subclass.

get_archive()

Get the archive for compressed files.

Raises:: NotImplementedError – If not implemented by subclass

classmethod delete_archive(full_path)

Deletes the archive.

This must be implemented in subclass to handle the specifics for different storage backends.

Parameters:: full_path (str) – Full path to the stored archive.
Returns:: True if deleted, else False.
Return type:: (boolean)

class devilry.devilry_compressionutil.backends.backends_base.PythonZipFileBackend(**kwargs)

Defines a baseclass backend using ZipFile for the Registry.

This class should be subclassed by backend-specific classes(backends for Heroku, S3, etc).

Parameters:

archive_path – Full path to archive including the archive name.
archive_name – Only the archive name.
readmode – Can be read from, defaults to True.

add_file(path, filelike_obj)

Add files to archive.

Parameters:

path (str) – Path to the file inside the Zip-archive.
filelike_obj – An object with method read()

Raises:

ValueError – If readmode is set to True, must be False to add files.

read_archive()

Get the zipped archive as ZipFile in readmode.

Returns:: The zipped archive.
Return type:: ZipFile

class devilry.devilry_compressionutil.backends.backends_base.PythonTarFileBackend(stream=False, compression='', **kwargs)

A baseclass backend using TarFile for archiving files to at tarball. Supports no compression, gzip and``bzip2`` compression through the :class:’~TarFile’ class.

Parameters:

stream – If it should be handled as a stream or not.
compression – The compression mode used, defaults to uncompressed, but modes gz, bz2 and xz can also be used.

compression_formats = ['', 'gz', 'bz2']: Compression formats supported.

add_file(path, filelike_obj)

Writes a file to the archive on the given path.

In fact, this creates a temporary folder hierarchy which is zipped when closed() is invoked on a instance of this class.

Parameters:

path – Path to file inside the archive.
filelike_obj – An object that behaves like a File(read, write..).

read_archive()

Get TarFile in readmode.

Returns:: The compressed tar archive.
Return type:: TarFile

close()

Must be invoked in order to finalize the archive by compressing the ‘temp’ directory to whichever compression level specified.

The temporary directory is then removed, and the tar archive is closed.

get_archive()

Get TarFile(archive) in read-mode.

Returns:: Opened in read mode.
Return type:: TarFile

devilry_compressionutil — Devilry utils models and backends

About the devilry compressionutil app

How to define a backend

Datamodel API for caching metainfo about a archive

Backend base classes

`devilry_compressionutil` — Devilry utils models and backends