devilry_compressionutil — Devilry utils models and backends

The devilry_compressionutil handles the creation of compressed archives.

About the devilry compressionutil app

The devilry_compressionutil app provides utilities for creating and adding archives using ZipUtil, by providing a backend and a registry for the specialized backends to use.

devilry_compressionutil also provides a meta class used for caching metainfo about a archive.

How to define a backend

Add path setting

First off, add a path to the storage location Django settings config. Call this setting DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY

DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY = os.path.join('path', 'to', 'storage', 'location')

Subclass backend

Create a backend for your app that you will use to compress the files by subclassing e.g PythonZipFileBackend in devilry.devilry_compressionutil.backends.backends_base.

from devilry.devilry_compressionutil.backends import backends_base

class YourAppZipBackend(backends_base.PythonZipFileBackend):
    backend_id = 'some_backend_id'

    def __init__(self, **kwargs):
        super(YourAppZipBackend, self).__init__(**kwargs)

Load backend to registry

Create an AppConfig for your app. Load the backend you created to the devilry_compressionutil s registry. Here you can register as many backends as you want, and just fetch it by its backend_id. The registry is a singleton, and to add a backend to it Registry.get_instance().add(..) must be called.

...
from devilry.devilry_compressionutil import backend_registry

class YourAppAppConfig(AppConfig):
    ...

    def ready(self):
        from devilry.your_app import your_backends
        backend_registry.Registry.get_instance().add(your_backends.YourAppZipBackend)

Get backend and add folders and files to it

Add paths and files to the backend.

from devilry.devilry_compressionutil import backend_registry

class HandleZipping:
    ...

    def your_function_that_uses_zip_backend(...):
        backend_class = backend_registry.Registry.get_instance().get('some_backend_id')

        # Path to archive inside the location specified by the storage location defined in
        # settings. ``DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY``
        archive_name = 'archive_name'
        full_archive_path = os.path.join('path', 'to', 'archive', 'archive_name')

        # Instance the backend.
        backend_instance = backend_class(
            archive_path=full_archive_path,
            archive_name=archive_name,
            readmode=False
         )

        # Get files to compress
        file1 = open('examplefile1.txt', 'r')
        file2 = open('examplefile2.txt', 'r')

        # Add file1 inside a folder named folder1
        # Add file2 inside a folder named folder2
        # The archive with content will now look like this
        # archive_name.zip
        #     - folder1
        #         - examplefile1.txt
        #     - folder2
        #         - examplefile2.txt
        backend_instance.add_file(os.path.join('folder1', 'examplefile1.txt'), file1)
        backend_instance.add_file(os.path.join('folder2', 'examplefile2.txt'), file2)

        # Close the backend
        backend_instance.close()

        # Read the archive as a fileobject
        # ``read_archive()`` returns a FileObject of the archive
        # This can for instance be passed to a HttpRequest.
        backend_instance.readmode = True
        fileobj = backend_instance.read_archive()

Datamodel API for caching metainfo about a archive

class devilry.devilry_compressionutil.models.GenericMeta(*args, **kwargs)

Bases: django.db.models.base.Model

Abstract class that implements usage of GenericForeignKey.

content_type

Foreignkey to Djangos ContentType.

content_object_id

ID of model to store.

content_object

An arbitrary model to store.

class devilry.devilry_compressionutil.models.CompressedArchiveMetaQueryset(model=None, query=None, using=None, hints=None)

Bases: django.db.models.query.QuerySet

Manager for class CompressedArchiveMeta.

create_meta(instance, zipfile_backend, user, user_role='')

Manager provides a way to create a meta entry for a archive. See CompressedArchiveMeta.

Parameters:
  • instance – Instance the archive is for.
  • zipfile_backend – base backend for compression, see PythonZipFileBackend.
delete_compressed_archives_older_than(days=None, seconds=None)

Delete compressed archive meta entries older than a specified number of days or seconds.

Parameters:
  • older_than_days (int) – Delete everything older than this.
  • older_than_seconds (int) – Delete everything older than this.
delete_compressed_archives_marked_as_deleted()

Delete all compressed archives marked as deleted.

class devilry.devilry_compressionutil.models.CompressedArchiveMeta(*args, **kwargs)

Bases: devilry.devilry_compressionutil.models.GenericMeta

Metadata about a compressed archive. Name of the archive, path to it and it’s size.

created_by

Who created the archive.

CREATED_BY_ROLE_STUDENT = 'student'

Use this as value for user_role if the user is commenting as a student.

CREATED_BY_ROLE_EXAMINER = 'examiner'

Use this as value for user_role if the user is commenting as an examiner.

CREATED_BY_ROLE_ADMIN = 'admin'

Use this as value for user_role if the user is commenting as an admin.

CREATED_BY_ROLE_CHOICES = (('student', u'Student'), ('examiner', u'Examiner'), ('admin', u'Admin'))

Choices for the user_role field.

created_by_role

What role did the user create the archive with?

created_datetime

When the archive was created.

archive_name

The actual name of the archive, Example.SomeArchive2000.zip.

archive_path

Path at storage location of the compressed archive. Example: https://s3-eu-central-1.amazonaws.com/BUCKET/path/to/archive/SomeArchive2000.zip

archive_size

Size of the archive in bytes.

backend_id

The ID of the backend used. This is the ID attribute backend_id.

deleted_datetime

When the entry was marked for deletion.

exception DoesNotExist

Bases: django.core.exceptions.ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: django.core.exceptions.MultipleObjectsReturned

Backend base classes

class devilry.devilry_compressionutil.backends.backends_base.BaseArchiveBackend(archive_path, archive_name='', readmode=True)

Bases: object

Specifies the interface for a backend compression-subclass.

All backends must implement this class.

Parameters:
  • archive_path – Full path to archive including the archive name.
  • archive_name – Only the archive name.
  • readmode – Can be read from, defaults to True.
backend_id = None

A unique string ID for the subclasses to use that describes what kind of backend it is.

get_storage_location()

Get the storage location for archives.

Returns:Location specified in settings DEVILRY_COMPRESSED_ARCHIVES_DIRECTORY.
Return type:str
read_binary()

Opens archive in read binary mode. Best suited for non-text files like images, videos etc or when serving file for download.

Returns:file object in read binary mode.
Return type:file
Raises:ValueError – If archive is None, or readmode is False.
close()

Close archive when done with adding files to it.

Raises:ValueError – If archive is None.
archive_size()

Get size of archive. Uses os.stat.

Returns:size of archive.
Return type:int
Raises:ValueError – If not in readmode or archive is None.
add_file(path, filelike_obj)

Add file to archive.

Parameters:
  • path (str) – Path to the file inside the archive.
  • filelike_obj – An object which implements function read().
Raises:

NotImplementedError – If not implemented by subclass.

read_archive()

Should return a object of the underlying compression tool in readmode.

Raises:NotImplementedError – If not implemented by subclass.
get_archive()

Get the archive for compressed files.

Raises:NotImplementedError – If not implemented by subclass
classmethod delete_archive(full_path)

Deletes the archive.

This must be implemented in subclass to handle the specifics for different storage backends.

Parameters:full_path (str) – Full path to the stored archive.
Returns:True if deleted, else False.
Return type:(boolean)
class devilry.devilry_compressionutil.backends.backends_base.PythonZipFileBackend(**kwargs)

Bases: devilry.devilry_compressionutil.backends.backends_base.BaseArchiveBackend

Defines a baseclass backend using ZipFile for the Registry.

This class should be subclassed by backend-specific classes(backends for Heroku, S3, etc).

add_file(path, filelike_obj)

Add files to archive.

Parameters:
  • path (str) – Path to the file inside the Zip-archive.
  • filelike_obj – An object with method read()
Raises:

ValueError – If readmode is set to True, must be False to add files.

read_archive()

Get the zipped archive as ZipFile in readmode.

Returns:The zipped archive.
Return type:ZipFile
class devilry.devilry_compressionutil.backends.backends_base.PythonTarFileBackend(stream=False, compression='', **kwargs)

Bases: devilry.devilry_compressionutil.backends.backends_base.BaseArchiveBackend

A baseclass backend using TarFile for archiving files to at tarball. Supports no compression, gzip and``bzip2`` compression through the :class:’~TarFile’ class.

Parameters:
  • stream – If it should be handled as a stream or not.
  • compression – The compression mode used, defaults to uncompressed, but modes gz, bz2 and xz can also be used.
compression_formats = ['', 'gz', 'bz2']

Compression formats supported.

add_file(path, filelike_obj)

Writes a file to the archive on the given path.

In fact, this creates a temporary folder hierarchy which is zipped when closed() is invoked on a instance of this class.

Parameters:
  • path – Path to file inside the archive.
  • filelike_obj – An object that behaves like a File(read, write..).
read_archive()

Get TarFile in readmode.

Returns:The compressed tar archive.
Return type:TarFile
close()

Must be invoked in order to finalize the archive by compressing the ‘temp’ directory to whichever compression level specified.

The temporary directory is then removed, and the tar archive is closed.

get_archive()

Get TarFile(archive) in read-mode.

Returns:Opened in read mode.
Return type:TarFile