angelos.archive7.streams¶

Data streams.

class angelos.archive7.streams.BaseStream(manager: StreamManager, block: angelos.archive7.streams.StreamBlock, identity: uuid.UUID, begin: int = - 1, end: int = - 1, count: int = 0, length: int = 0, compression: int = 0)¶

Bases: object

Descriptor for an open data stream to be read and written too.

Data streams should be wrapped in a file descriptor.

self.__identity. unsigned integer, the id number of the stream in the registry. self.__begin. signed integer, position of the first block in the stream. self.__end. signed integer, position of the last block in the stream. self.__count. unsigned integer, number of blocks used. self.__length. unsigned long long, number of bytes in the data stream. self.__compression, unsigned short, compression algorithm of choice.

COMP_NONE = 0¶

FORMAT = '!16siiIQH'¶

SIZE = 38¶

property block¶: Expose current block.

changed()¶: Indicate that the current block has been written to.

abstract close()¶: Save and close stream at manager.

property count¶: Expose block count.

property data¶: Expose the current block’s data section.

end()¶: Forcefully wind to the end of stream.

extend() → bool¶

Create a new block at the end of stream.

Returns (bool):: True if successfully created a new block or False if not at end of stream.

property identity¶: Expose the streams identity number.

property len¶: Expose stream byte length.

length(change: int = 0) → int¶

Tell and update the length of the stream in used bytes.

Parameters: change (int) – A relative number of bytes added or subtracted.

Returns (int):: Current or new length of stream.

load_meta(stream: Union[bytearray, bytes]) → bool¶

Unpack metadata and populate the fields.

Parameters: stream (Union[bytearray, bytes]) – Bytes to be read into metadata.

Returns (bool):: True if the loaded data is not corrupt else False

property manager¶: Expose stream manager.

static meta_unpack(data: Union[bytes, bytearray])¶

Parameters: data (Union[bytes, bytearray]) – Data to be unpacked

Returns (tuple):: Unpacked tuple

next() → bool¶

Load next data block in the stream.

Returns (bool):: True if next block loads, if it is last block False.

pop() → angelos.archive7.streams.StreamBlock ¶

previous() → bool¶

Load previous data block in the stream.

Returns (bool):: True if previous block loads, if it is last block False.

push(block: angelos.archive7.streams.StreamBlock)¶

save(enforce: bool = False)¶

Save current block if changed or enforced.

Parameters: enforce (bool) – Enforce writing the block to disk.

truncate(length: int) → int¶

Truncate the stream to a certain length and recycle blocks.

Parameters: length (int) – New length of stream.

Returns (int):: New length of stream.

wind(index: int) → int¶

Wind forward or backward to block index.

Parameters: index (int) – Block index to wind to.

Returns (int):: New block index.

class angelos.archive7.streams.BaseStreamManager¶

Bases: abc.ABC

BLOCK_META = 0¶

SPECIAL_BLOCK_COUNT = 0¶

SPECIAL_STREAM_COUNT = 0¶

abstract close()¶

property closed¶

property created¶

abstract load_block(index: int) → angelos.archive7.streams.StreamBlock ¶

property meta¶

abstract new_block() → angelos.archive7.streams.StreamBlock ¶

abstract recycle(chain: angelos.archive7.streams.StreamBlock) → bool¶

abstract reuse() → angelos.archive7.streams.StreamBlock ¶

abstract save_block(index: int, block: angelos.archive7.streams.StreamBlock)¶

abstract save_meta()¶

abstract special_block(position: int)¶

abstract special_stream(position: int)¶

class angelos.archive7.streams.DataStream(manager: StreamManager, block: angelos.archive7.streams.StreamBlock, identity: uuid.UUID, begin: int = - 1, end: int = - 1, count: int = 0, length: int = 0, compression: int = 0)¶

Bases: angelos.archive7.streams.BaseStream

Stream for general use.

close()¶: Save block and close stream at manager.

class angelos.archive7.streams.DynamicMultiStreamManager(filename: pathlib.Path, secret: bytes)¶

Bases: angelos.archive7.streams.FixedMultiStreamManager

Stream manager handles all the streams and blocks that are underlying of a virtual file system.

The underlying system is built up of 4Kb blocks that can be chained like linked lists, those are data streams. There can be and are several data streams, they can be used for files and can expand by adding more blocks, thanks to this streams can grow in size.

There are reserved blocks and streams. In total the first eight blocks are reserved, so are the first eight streams.

SPECIAL_STREAM_COUNT = 2¶

STREAM_INDEX = 1¶

close_stream(stream: angelos.archive7.streams.DataStream) → bool¶

Close an open data stream.

Parameters: stream (DataStream) – Data stream object being saved.

del_stream(identity: uuid.UUID) → bool¶

Delete data stream from file.

Parameters: identity (uuid.UUID) – Data stream number to be erased.

Returns (bool):: Success of deleting data stream.

new_stream() → angelos.archive7.streams.DataStream ¶

Create a new data stream.

Returns (DataStream):: The new data stream created.

open_stream(identity: uuid.UUID) → angelos.archive7.streams.DataStream ¶

Open an existing data stream.

Parameters: identity (uuid.UUID) – Data stream number.

Returns (DataStream):: The opened data stream object.

class angelos.archive7.streams.FixedMultiStreamManager(filename: pathlib.Path, secret: bytes)¶

Bases: angelos.archive7.streams.StreamManager

Multi stream manager, that manages a fixed set of streams.

SPECIAL_BLOCK_COUNT = 1¶

SPECIAL_STREAM_COUNT = 1¶

STREAM_TRASH = 0¶

recycle(block: angelos.archive7.streams.StreamBlock)¶

Recycle a truncated chain of blocks from a stream.

Parameters: chain (StreamBlock) – Block and chain that will be recycled

reuse() → angelos.archive7.streams.StreamBlock ¶

Get a recycled block if any available.

Returns (StreamBlock):: Block to be reused or None

class angelos.archive7.streams.InternalStream(manager: StreamManager, block: angelos.archive7.streams.StreamBlock, identity: uuid.UUID, begin: int = - 1, end: int = - 1, count: int = 0, length: int = 0, compression: int = 0)¶

Bases: angelos.archive7.streams.BaseStream

Stream for internal use in StreamManager.

close()¶: Save block.

class angelos.archive7.streams.Registry(manager: DynamicMultiStreamManager)¶

Bases: abc.ABC

B+Tree registry and wal wrapper

close()¶

property tree¶

class angelos.archive7.streams.SingleStreamManager(filename: pathlib.Path, secret: bytes)¶

Bases: angelos.archive7.streams.StreamManager

SPECIAL_BLOCK_COUNT = 1¶

SPECIAL_STREAM_COUNT = 1¶

STREAM_DATA = 0¶

recycle(chain: angelos.archive7.streams.StreamBlock) → bool¶: Truncate stream at block position.

reuse() → angelos.archive7.streams.StreamBlock ¶: Single stream don’t need reuse.

class angelos.archive7.streams.StreamBlock(position: int, previous: int = - 1, next: int = - 1, index: int = 0, stream: uuid.UUID = UUID('00000000-0000-0000-0000-000000000000'), block: bytes = None)¶

Bases: object

A block of data in a stream.

The amount of raw data is set to 4020 bytes, except 16 bytes for metadata and 20 bytes of digest. This sums up to 4056 bytes, after encryption and its digest we end up with 4096 bytes or 4 Kb.

self.previous. 4 bytes, signed integer linking to previous block. self.next. 4 bytes, signed integer linking to next block. self.index. 4 bytes, unsigned integer block in stream index. self.stream. 16 bytes, unsigned integer setting stream id. self.digest. 20 bytes, sha1 digest of the data field. self.data. 4004 bytes

FORMAT = '!iiI16s20s4008s'¶

SIZE = 4056¶

data¶

digest¶

index¶

load_meta(block: bytes)¶

Unpack a block of bytes into its components and populate the fields

Parameters: block (bytes) – Bytes to be read into a data block.

Returns (bool):: True if the loaded data is not corrupt else False

static meta_unpack(data: Union[bytes, bytearray])¶

Parameters: data (Union[bytes, bytearray]) – Data to be unpacked

Returns (tuple):: Unpacked tuple

next¶

property position¶: Expose stream block position in file.

previous¶

stream¶

class angelos.archive7.streams.StreamManager(filename: pathlib.Path, secret: bytes)¶

Bases: abc.ABC

Stream manager handles streams with their blocks and provides transparent encryption.

Transparent encryption is built in including standards for carrying out different kind of operations on the streams and blocks.

BLOCK_META = 0¶

SPECIAL_BLOCK_COUNT = 0¶

SPECIAL_STREAM_COUNT = 0¶

close()¶

property closed¶

property created¶

load_block(index: int) → angelos.archive7.streams.StreamBlock ¶

Load a block from index and decrypt.

Parameters: index (int) – Block index.

Returns (StreamBlock):: Loaded block a stream block.

property meta¶

new_block() → angelos.archive7.streams.StreamBlock ¶

Create new block at the end of file, write empty block to file.

Returns (StreamBlock):: The newly created block.

abstract recycle(chain: angelos.archive7.streams.StreamBlock) → bool¶

abstract reuse() → angelos.archive7.streams.StreamBlock ¶

save_block(index: int, block: angelos.archive7.streams.StreamBlock)¶

Save a block and encrypt it.

Parameters

index (int) – Index for offset where to write block
block (StreamBlock) – Block to save to file.

save_meta()¶: Save meta information.

special_block(position: int)¶: Receive one of the 8 reserved special blocks.

special_stream(position: int)¶: Receive one of the 3 reserved special streams.

class angelos.archive7.streams.StreamRegistry(manager: DynamicMultiStreamManager)¶

Bases: angelos.archive7.streams.Registry

Registry to keep track of all streams and the trash.

register(stream: angelos.archive7.streams.DataStream) → int¶

Register a data stream.

Parameters: stream (DataStream) – Stream to be registered.

Returns (int):: Stream identity number.

search(identity: int) → bytes¶

Search for a stream by identity number.

Parameters: identity (int) – Identity number.

Returns (bytes):: Stream metadata.

unregister(identity: int) → bytes¶

Unregister a data stream.

Parameters: identity (int) – Stream identity number.

Returns (bytes):: Stream metadata.

update(stream: angelos.archive7.streams.DataStream) → bool¶

Update stream metadata.

Parameters: stream (DataStream) – Stream to update metadata from.

Returns (bool):: Success of update.

class angelos.archive7.streams.VirtualFileObject(stream: angelos.archive7.streams.DataStream, filename: unicode, mode: unicode = 'r')¶

Bases: angelos.archive7.base.BaseFileObject

Stream for the registry index database.

property stream¶: Expose the internal data stream.