====================================================
A video frame in transcode -- Birth, life and death.
====================================================

This article has information about the generic stages of a frame in
the transcode core. It may help you when you want to do a filter for
transcode or just understand how transcode works.

Transcode is a threaded application. The different threads which are
of importance here are one decoder thread, one encoder thread and N
frame processing threads. N is a number the user can control via
the second argument of transcodes -u option.

Each frame is put into a frame_list_t (/src/framebuffer.h)
structure. The frames are buffered internally in a double linked
list. The order of an frame in this list denotes the decoding (and
therefore) the encoding order (/src/video_buffer.c)

The import module decompresses a frame from the input file into a
format the core of transcode likes. Just after the frame arrives in
the decoder, the status of the frame is set to FRAME_WAIT.
the first frame pre-processing takes place here.

1) preprocess_vid_frame() (/src/frame_preprocess.c)
   If the frame number is out-of-range as denoted by -c, the skipped
   attribute is attached to the frame and it is ignored by further
   processing routines. If the user has given --pre_clip, this will
   happen here.

After the very early clipping the filters get their draw with the tag:

2) TC_PRE_S_PROCESS
   The _S_ stands for synchronous. This means for the filter it will
   recieve one frame after another in decoding order. The filter is
   executed by the decoding thread and runs inside this thread
   putting the decoder on halt for its execution time.

When all filters are done with TC_PRE_S_PROCESS, the frame leaves
the decoder and the sychronous part and now waits to be picked up by
one of the frame processing threads (/src/frame_threads.c). One
frame processing thread does the following: It grabs one frame from
the frame_list and sets its state to FRAME_LOCK to make sure no
other frame processing thread will take this frame. Now the frame
enters the filters again:

3) TC_PRE_M_PROCESS
   The frame passes all the filters in the order specified on the
   command line. The frame ordering is not garanteed any more; the
   frames may arrive in the filter in any order. The filter must be
   reentrant in some way.

Next, the internal processing routines take place:

4) process_vid_frame() (/src/video_trans.c)
   The following operations are applied to the frame in the hereby
   given order
   -j (clip) -I (deinterlace) -X (fast scale up) -B (fast scale
   down) -Z (slow zoom) -Y (clip) -r (scale half) -z (flip)
   -l (mirror) -k (rgbswap) -K (make grayscale) -G (gamma)
   -C (antialias)
   short: "jIXBZYrzlkKGC"

The frame wanders off into the filters a third time:

5) TC_POST_M_PROCESS
   (the same as for 3) applies here, too)

The frame processing thread now sets the status of the frame to
FRAME_READY which means the frame can now be picked up by the
encoder. The encoder does again some processing and calls the
filters for the fourth and last time.

6) TC_POST_S_PROCESS
   (the same as for 2) applies here, too)

After this a internal post processing takes place:

7) postprocess_vid_frame() (/src/frame_postprocess.c)
   Very last clipping if the user wants this via --post_clip

Finally, the processed frame is made available to preview filters with
the tag:

8) TC_PREVIEW
   This tag is intended only for the "preview" and "pv" filters (and any
   new ones created with the same purpose), and allows the final video
   frame after clipping via --post_clip to be shown to the user.  The
   frame should not be modified here.

When the frame has the skip flag set it will not be encoded, all
other frames will encoded and freed.


===================
Graphical structure
===================

                         decode()
                            |
                     1) preprocess()
                            |
               2) filters(TC_PRE_S_PROCESS)
                            |
             _____________/ | \______________
            /      |        |        |       \
           /       |        |        |        \
     N  F R A M E  P R O C E S S I N G  T H R E A D S
        /          .        .        .         .
       |           .        .        .         .
       |
 3) filters(TC_PRE_M_PROCESS)
       |
 4) process_vid_frame()
       |
 5) filters(TC_POST_M_PROCESS)
       |
       |           .        .        .         .
        \          .        .        .         .
         \         |        |        |        /
          \________|______  |  ______|_______/
                          \ | /
                            |
             6) filters(TC_POST_S_PROCESS)
                            |
                   7) postprocess()
                            |
                 8) filters(TC_PREVIEW)
                            |
                         encode()


==============
Cloning Frames
==============

A filter can clone a frame. What this means and when it is possible
to do it will be the topic of this section. We only consider the
filters slots since non of the core video function does cloning.
"Cloning a frame" means a frame gets duplicated (cloned) and encoded
twice. Of course, if the cloned frame gets modified, a different
cloned frame gets encoded. As the reader might know, there are four
filter slots in transcode (excluding TC_PREVIEW, which is not
relevant to this discussion). I'll abbrevate them for easier reading:

 Slot                 | Abb. | Comment
 ---------------------+------+----------------------------------
 TC_PRE_S_PROCESS     |  ES  | synchronous pre processing
 TC_PRE_M_PROCESS     |  EM  | multithreaded pre processing
 TC_POST_M_PROCESS    |  OM  | multithreaded post processing
 TC_POST_S_PROCESS    |  OS  | synchronous post processing

Every filter in every slot can clone frames but the filter may not
be called again with the cloned frame again. This depends on the
slot the filter lives in. Note that a filter can live in several
slots, for example, it can set the clone flag at ES time and catch
the cloned frame at OM time if it likes to do so. The filter must
keep in mind, that the cloned frame may have different geometry at
OM time because it may be rescaled or resampled.

How to read the following table. The field "Slot" denotes the slot
the filter is in when it sets CLONE. The "Slots to pass" field tells
the slots the cloned frame will pass.

 Slot  | Slots to pass | Slots NOT passed
 ------+---------------+-----------------
 ES    | EM OM OS      | ES
 EM    | EM OM OS      | ES
 OM    |       OS      | ES EM OM
 OS    |       OS      | ES EM OM

Example 1:
A filter in ES sets CLONE. The filter itself will not see the cloned
frame, but all filters in following slots will.

Example 2:
A filter in EM sets CLONE. The filter _will_ see the cloned frame
again and all consecutive slots will get it again, too.

Notes on Cloning.
To clone a frame, the filter adds TC_FRAME_IS_CLONED to the frame
pointer attributes. The frame ID never gets incremented to reflect
the count of cloned frames. If the filter decides to clone a frame,
it gets the frame back with the same ID but with TC_FRAME_WAS_CLONED
set, so its easy to distinguish between the frame with the original
ID and the duplicated ID.


===============
Skipping Frames
===============

A filter can apply the skipped attribute to a frame which tells
transcode the frame is not to be encoded but dropped. Not all slots
can skip a frame; only slots where there is asynchronous A/V are
allowed to do so.

 Slot  | slot in which the frame will be dropped
 ------+----------------------------------------
 ES    |  Before EM
 EM    |  After EM, before OM
 OM    |  After OM, before OS
 OS    |  After OS, before encode

In other words, a skipped frame gets dropped right after the filter
has returned.

=================================
Choosing the Slot for your Filter
=================================

If you want to do a filter you may not be sure in which slot it
should be in. This section helps you in finding the correct slot for
your filter.

Multithreaded vs. Synchronous

 You should try as hard as possible to put your filter into one of
 the _M_ slots. An _M_ slot is usually faster than an _S_ slot.
 Imagine the decoder waits for data to arrive from the harddrive and
 the encoder is busy. Having your filter run in _M_ could use the
 time while noone else is doing something to process the frame.
 The _S_ stages are run directly by the decoder respectivly the
 encoder and will block the de/encoder in doing their real work.

 However, its not always possible to put a filter in _M_. In _M_
 slots the filter may recieve the frames in any order. So if the
 filter does depend on correct frame ordering it cannot be in _M_.
 It may be possible to rewrite the filter in a way so that it does
 not depend on the order of frames, check if it can be done for your
 filter.

PRE vs. POST

 This decision is merely driven by the fact what your filter
 actually does and which kind of data you want to have. Its usually
 true, that a POST filter has to deal with less data than a PRE
 filter. Of course, this is only true, if the user scales down and
 not up but this is what most users do.

 As an example, a deinterlace should always be run as a PRE filter
 before the the up/down scaling of the frame happens in
 transcode-core. A denoiser can be either run in PRE or in POST
 because it just does not matter. One can assume that if the user
 wants deinterlacing, in POST you can be pretty sure that you work
 with progressive material.

Always keep in mind that a filter can be in multiple slots. It
depends on the filter which tag it wants.


=================================
Multiple Instances of your Filter
=================================

If your you want to be able to run XXX: see /filter/filter_32detect.c

=======================
Linked Framebuffer List
=======================
// XXX: WRITEME

// vim: tw=68
