Python API¶

Structure¶

class destructify.Structure(_context=None, **kwargs)¶

You use Structure as the base class for the definition of your structures. It is a class with a metaclass of StructureBase that enables the fields to be parsed separately.

len(Structure): This is a class method that allows you to retrieve the size of the structure, if possible.

classmethod from_stream(stream, context=None)¶

Reads a stream and converts it to a Structure instance. You can explicitly provide a ParsingContext, otherwise one will be created automatically.

This will seek over the stream if one of the alignment options is set, e.g. ParsingContext.alignment or Field.offset. The return value in this case is the difference between the start offset of the stream and the offset of the highest read byte. In most cases, this will simply equal the amount of bytes consumed from the stream.

Parameters:	stream – A buffered bytes stream. context (ParsingContext) – A context to use while parsing the stream.
Return type:	Structure, int
Returns:	A tuple of the constructed `Structure` and the amount of bytes read (defined as the last position of the read bytes).

classmethod from_bytes(bytes, context=None)¶: A short-hand method of calling from_stream(), using bytes rather than a stream, and returns the constructed Structure immediately.

classmethod initialize(context)¶

This classmethod allows you to modify the ParsingContext, just after all values were read from the stream and Field.get_initial_value() was called, but before the Structure is created. This can be used to modify some values of the structure just before it is being created.

Parameters:	context (ParsingContext) – The context of the initializer

to_stream(stream, context=None)¶

Writes the current Structure to the provided stream. You can explicitly provide a ParsingContext, otherwise one will be created automatically.

This will seek over the stream if one of the alignment options is set, e.g. ParsingContext.alignment or Field.offset. The return value in this case is the difference between the start offset of the stream and the offset of the highest written byte. In most cases, this will simply equal the amount of bytes written to the stream.

Parameters:	stream – A buffered bytes stream. context (ParsingContext) – A context to use while writing the stream.
Return type:	int
Returns:	The number bytes written to the stream (defined as the maximum position of the bytes that were written)

to_bytes(context=None)¶: A short-hand method of calling to_stream(), writing to bytes rather than to a stream. It returns the constructed bytes immediately.

finalize(context)¶

Function that allows for modifying the ParsingContext just after filling the context with the values obtained by Field.get_final_value(), before it will be converted to binary data. This can be used to modify some values of the structure just before it is being written, e.g. for checksums.

Parameters:	context (ParsingContext) – The context of the finalizer

__bytes__()¶: Same as to_bytes(), allowing you to use bytes(structure)

classmethod as_cstruct()¶

_meta¶: This allows you to access the StructureOptions class of this Structure.

_context¶: If this Structure was created by from_stream(), this contains the ParsingContext that was used during the processing. Otherwise, this attribute is undefined.

Field¶

class destructify.Field(*, name=None, default=NOT_PROVIDED, override=NOT_PROVIDED, decoder=None, encoder=None, offset=None, skip=None, lazy=False)¶

A basic field is incapable of parsing or writing anything, as it is intended to be subclassed.

ctype¶: A friendly description of the field in the form of a C-style struct definition.

preparsable¶

Indicates whether this field is preparsable. This is used to indicate that a field can be parsed outside of the normal sequential loading, allowing referencing this field before it is defined.

The simple implementation is that the field is lazy and has an absolute offset set.

full_name¶: The full name of this Field.

field_context¶

The FieldContext that is used in the ParsingContext for this field. It returns a partially resolved function call with the current field already set.

Return type:	type

preparsable

Indicates whether this field is preparsable. This is used to indicate that a field can be parsed outside of the normal sequential loading, allowing referencing this field before it is defined.

The simple implementation is that the field is lazy and has an absolute offset set.

stream_wrappers¶: Returns an iterable of classes that are required stream wrappers on a Structure level. For instance, this may return a list containing a BitStream to enable bit-based methods.

with_name(name)¶: Context manager that yields this Field with a different name. If name is None, this is ignored.

A Field also defines the following methods:

len(field)

You can call len on a field to retrieve its byte length. It can either return a value that makes sense, or it will raise an ImpossibleToCalculateLengthError when the length depends on something that is not known yet.

Some attributes may affect the length of the structure, while they do not affect the length of the field. This includes attributes such as skip. These are automatically added when the structure sums up all fields.

If you need to override how the structure sums the length of fields, you can override _length_sum. You must then manually also include those offsets. This is only used by BitField.

initialize()¶: Hook that is called after all fields on a structure are loaded, so some additional multi-field things can be arranged.

get_initial_value(value, context)¶

Returns the initial value given a context. This is used by Structure.from_stream() to retrieve the value that is read from the stream. It is called after all fields have been parsed, so inter-field dependencies can be resolved here.

The value may be a proxy object if lazy is set.

Parameters:	value – The value to retrieve the final value for. context (ParsingContext) – The context of this field.

get_final_value(value, context)¶

Returns the final value given a context. This is used by Structure.to_stream() to retrieve the value that is to be written to the stream. It is called before any fields have been processed, so inter-field dependencies can be resolved here.

Parameters:	value – The value to retrieve the final value for. context (ParsingContext) – The context of this field.

seek_start(stream, context, offset)¶

This is called before the field is parsed/written. It should expect the stream to be aligned to the ending of the previous field. It is intended to seek its starting position. This makes sense if the offset is set, for instance. In the case this stream is not tellable and no seek is performed, offset is returned unmodified.

Note that the relative offset is passed in, but the absolute offset is expected as a result.

Parameters:	stream (io.BufferedIOBase) – The IO stream to consume from. context (ParsingContext) – The context used for the parsing. offset (int) – The current relative offset in the stream
Returns:	The new absolute offset in the stream

seek_end(stream, context, offset)¶

This is called when the field is lazy and we need to find the end of the field. This is not called when the field is actually read, as from_stream() is expected to align to the end of the field.

This method should be as efficient as possible with retrieving the length. For instance, if it is possible to read a few bytes and then determine how long this field is, that is fine. If it is not possible without reading the entire field, this method should return None.

The default implementation is to call len(self) and use that if possible.

Parameters:	stream (io.BufferedIOBase) – The IO stream to consume from. context (ParsingContext) – The context used for the parsing. offset (int) – The current relative offset in the stream
Returns:	The new absolute offset in the stream, or None if this field can not be processed without parsing it entirely.

decode_value(value, context)¶

This value is called just after the value is retrieved from from_stream(). It should return an adjusted value that is the true representation of the value

Parameters:	value – The value to retrieve the decoded value for. context (ParsingContext) – The context of this field.

encode_value(value, context)¶

This value is called just before the value is passed to to_stream(). It should return an adjusted value that is accepted by to_stream(). This is typically used in conjunction with encoder.

Parameters:	value – The value to retrieve the encoded value for. context (ParsingContext) – The context of this field.

from_stream(stream, context)¶

Given a stream of bytes object, consumes a given bytes object to Python representation. The given stream is already at the start of the field. This method must ensure that the stream is after the end position of the field after reading. In other words, the following will typically hold true:

stream_at_start.tell() + result[1] == stream_at_end.tell()

The default implementation is to raise a NotImplementedError and subclasses must override this function.

Parameters:	stream (io.BufferedIOBase) – The IO stream to consume from. The current position is already set to the start position of the field. context (ParsingContext) – The context of this field.
Returns:	a tuple: the parsed value in its Python representation, and the amount of consumed bytes

to_stream(stream, value, context)¶

Writes a value to the stream, and returns the amount of bytes written. The given stream will already be at the start of the field, and this method must ensure that the stream cursor is after the end position of the field. In other words:

stream_at_start.tell() + result == stream_at_end.tell()

The default implementation is to raise a NotImplementedError and subclasses must override this function.

Parameters:	stream (io.BufferedIOBase) – The IO stream to write to. value – The value to write context (ParsingContext) – The context of this field.
Returns:	the amount of bytes written

decode_from_stream(stream, context)¶: Shortcut method to calling from_stream() and decode_value() in succession. Not intended to be overridden.

encode_to_stream(stream, value, context)¶: Shortcut method to calling encode_value() and to_stream() in succession. Not intended to be overridden.

ParsingContext¶

class destructify.ParsingContext(structure=None, *, parent=None, parent_field=None, flat=False, stream=None, capture_raw=False)¶

A context that is passed around to different methods during reading from and writing to a stream. It is used to contain context for the field that is being parsed.

While parsing, it is important to have some context; some fields depend on other fields during writing and during reading. The ParsingContext object is passed to several methods for this.

When using this module, you will get a ParsingContext when you define a property of a field that depends on another field. This is handled by storing all previously parsed fields in the context, or (if applicable) the Structure the field is part of. You can access this as follows:

context['field_name']

But, as a shorthand, you can also access it as an attribute of the f object:

context.f.field_name

context[key]: Returns the value of the specified key, either from the already parsed fields, or from the underlying structure, depending on the situation.

f¶

This object is typically used in lambda closures in Field declarations.

The f attribute allows you to access fields from this context, using attribute access. This is similar to using context[key], but provides a little bit cleaner syntax. This object is separated from the scope of ParsingContext to avoid any name collisions with field names. (For instance, a field named f would be impossible to reach otherwise).

f.name: Access the current value of the named field in the ParsingContext, equivalent to ParsingContext[name]

f[name]: Alias for attribute access to allow accessing names that are dynamic or collide with the namespace (see below)

Two attributes are offered for parent and root access, and a third one to access the ParsingContext. These names still collide with field names you may want to specify, but the f-object is guaranteed to not add any additional name collisions in minor releases.

f._¶

Returns the ParsingContext.f attribute of the ParsingContext.parent object, so you can write f.parent.parent.field, which is equivalent to context.parent.parent['field'].

If you need to access a field named _, you must use f['_']

f._root¶

Returns the ParsingContext.f attribute of the ParsingContext.root object, so you can write f.root.field, which is equivalent to context.root['field']

If you need to access a field named _root, you must use f['_root']

f._context¶

Returns the actual ParsingContext. Used in cases where a f-object is only provided.

If you need to access a field named _context, you must use f['_context']

parent¶: Access to the parent context (useful when parsing a Structure inside a Structure). May be None if this is the uppermost context.

parent_field¶: The field in the parent that is responsible for creation of this subcontext.

flat¶: Indicates that the parent context should be considered part of this context as well. This allows you to reference fields in both contexts transparently without the need of calling parent.

stream¶: The stream that is used during parsing.

capture_raw¶

Indicates whether FieldContext.raw should be filled. This is useful if you need to calculate values based on the raw contents of the field, for instance, for calculating checksums.

Note that this attribute cannot change the stream used by the parser. This means that the context cannot add the methods of the CaptureStream stream wrapper, which caches values written to and read from the stream, and must rely on a call to seek() on the stream to read the raw value.

It is recommended to use StructureOptions.capture_raw, as this can change the stream.

root¶: Retrieves the uppermost ParsingContext from this ParsingContext. May return itself.

fields¶: This is a dictionary of field names to FieldContext. You can use this to access information of how the fields were parsed. This is typically for debugging purposes, or displaying information about parsing structures.

done¶: Boolean indicating whether the parsing was done. If this is True, lazy fields can no longer become non-lazy.

field_values¶

Represents a immutable view on all field values from fields. This is highly inefficient if you only need to access a single value (use context[key]). The resulting dictionary is immutable.

This attribute is essentially only useful when constructing a new Structure where all field values are needed.

initialize_from_meta(meta, structure=None)¶: Adds fields to the context based on the provided StructureOptions. If structure is provided, the values in the structure are passed as values to the field contexts

When you are implementing a field yourself, you get a ParsingContext when reading from and writing to a stream.

FieldContext¶

class destructify.FieldContext(field, context, value=NOT_PROVIDED, *, field_name=None, parsed=False, offset=None, length=None, lazy=False, raw=None)¶

This class contains information about the parsing state of the specified field.

field¶: The field this FieldContext applies to.

field_name¶: If set, this is the name of the field that is used in the context, regardless of what field has as Field.name set. If this is set, this is used with Field.with_name() when parsing lazily.

value¶: The current value of the field. This only makes sense when has_value is True. This can be a proxy object if lazy is true.

has_value¶: Indicates whether this field has a value. This is true only if the value is set or when lazy is true.

parsed¶: Indicates whether this field has been written to or read from the stream. This is also true when lazy is true.

resolved¶: Indicates whether this fields no longer requires stream access, i.e. it is parsed and lazy is false.

offset¶: Indicates the offset in the stream of this field, relative to the parent of this field. Is only set when parsed is true.

absolute_offset¶: Returns the absolute offset in the stream of this field. Is only set when parsed is true.

length¶: Indicates the length of this field. Is normally set when parsed is true, but may be not set when lazy is true and the length was not required to be calculated.

lazy¶: Indicates whether this field is lazily loaded. When a lazy field is resolved during parsing of the structure, i.e. while ParsingContext.done is false, resolving this field will affect value, length and set lazy to false. After ParsingContext.done has become true, these attributes will not be updated.

raw¶: If ParsingContext.capture_raw is true, this field will contain the raw bytes of the field.

subcontext¶: This may be set if the field created a subcontext to parse its inner field(s).

add_parse_info(offset, length, value=NOT_PROVIDED, lazy=False)¶

Call that is used when the value has been parsed. This fills all information in te structure.

Parameters:	value – The value that has been parsed. offset – The offset of the value in the stream length – The length of the value in the stream lazy – Indicates whether the value is lazily loaded, i.e. the stream is not hit (value is ignored)