Obsidian-Main/03. 資料收集/FFMpeg.md

## Initialization
```cpp
av_register_all();
avformat_network_init();
```

## Command line
利用FFMpeg來轉檔
```
ffmpeg -i <SOURCE.mp4> -t 10 -s <W>x<H> -pix_fmt <OUTPUT_FORMAT> <OUTPUT_FILENAME>
```
- `-i`：來源檔名
- `-t`：影片長度（秒）
- `-s`：影片大小，長x寬
- `-pix_fmt`：輸出的影像格式

example:
```
ffmpeg -i test.mp4 -t 10 -s 240x128 -pix_fmt yuv420p out240x128.yuv
```

## Open file
使用`avformat_open_input()`來打開檔案。
```cpp
AVFormatContext* pFormatContext = NULL;
char* path = "E:\\download\\Incredible Patagonia_1920x1080.mp4";
AVDictionary* pDict = NULL;

int result = avformat_open_input(&pFormatContext, path, NULL, &pDict);
```
若是開啟失敗`result`會是一個非0的值。可以用`av_strerror()`來印出詳細的錯誤訊息。例：
```cpp
char errorMsg[1024] = { 0 };
av_strerror(result, errorMsg, sizeof(errorMsg) - 1);
printf("Open file %s error, code = %d, message = %s\n", path, result, errorMsg);
```

### Get file information
用`avformat_find_stream_info()`來把檔案的video/audio相關訊息放到`AVFormatContext`裡面。例：
```cpp
result = avformat_find_stream_info(pFormatContext, NULL);
```
也可以用
```cpp
av_dump_format(pFormatContext, 0, path, 0);
```
來直接印出相關訊息，不過這方法是Ffmpeg內定的，如果要印出自己想要的內容，就必須將`AVFormatContext`裡面的`AVStream`一一撈出來：
```cpp
for (int i = 0; i < pFormatContext->nb_streams; i++) {
    AVStream* avStream = pFormatContext->streams[i];

    if (avStream->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) {
        printf("Stream[%d] is a audio stream.\n", i);
        printf("    Sample rate: %d\n", avStream->codecpar->sample_rate);
        printf("    Format     : %d\n", avStream->codecpar->format);
        printf("    Channels   : %d\n", avStream->codecpar->channels);
        printf("    FPS        : %d/%d\n", avStream->avg_frame_rate.num, avStream->avg_frame_rate.den);
        printf("    Frame Size : %d\n", avStream->codecpar->frame_size);
        printf("    Codec ID   : %d\n", avStream->codecpar->codec_id);
    } else if (avStream->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) {
        printf("Stream[%d] is a video stream.\n", i);
        printf("    Width : %d\n", avStream->codecpar->width);
        printf("    Height: %d\n", avStream->codecpar->height);
        printf("    FPS   : %d/%d\n", avStream->avg_frame_rate.num, avStream->avg_frame_rate.den);
        printf("    Codec ID: %d\n", avStream->codecpar->codec_id);
    }
}
```
這個for-loop可以找出檔案裡面的video stream index與audio stream index。

另一個找出video stream index與audio stream index的方法：
```cpp
int videoStream = av_find_best_stream(pFormatContext, AVMEDIA_TYPE_VIDEO, -1, -1, NULL, 0);
int audioStream = av_find_best_stream(pFormatContext, AVMEDIA_TYPE_AUDIO, -1, -1, NULL, 0);
printf("Found best video stream index: %d\n", videoStream);
printf("Found best audio stream index: %d\n", audioStream);
```

## Demux
### 重要結構
#### AVPackt
##### 重要的member：
- `AVBufferRef *buf`：buffer的reference count，基本上不會用到。
- `int64_t pts`：顯示順序
- `int64_t dts`：decode順序
- `uint8_t *data`：實際存data的buffer
- `int size`：上面那個buffer的大小

##### 操作function
- `AVPacket* av_packet_alloc()`：產生一個兂的AVPacket
- `AVPacket* av_packet_clone(const AVPacket*)`：複製一個AVPacket
- `int av_packet_ref(AVPacket*, const AVPacket*)`：增加reference count
- `av_packet_unref(AVPacket*)`：減少reference count
- `void av_packet_free(AVPacket**)`：釋放AVPacket
- `void av_init_packet(AVPacket*)`：初始化AVPacket
- `int av_packet_from_data(AVPacket*, uint8_t*, int size)`：從自己的data 
- `av_seek_frame`：
    - `AVFormatContext*`
    - `int stream_index`：要seek的stream
    - `int64_t timestamp`：timestamp
    - `int flag`：seek的方法
        - `#define AVSEEK_FLAG_BACKWARD 1`：往後找
        - `#define AVSEEK_FLAG_BYTE     2`：以byte為單位，跳到timestamp相對應的byte
        - `#define AVSEEK_FLAG_ANY      4`：跳到任意frame，不一定要key frame
        - `#define AVSEEK_FLAG_FRAME    8`：依照frame number來找

## Decode
### 重要結構
#### AVCodecContext
##### 重要的member
- `int thread_count`：執行續的數量
- `AVRational time_base`

##### 操作function
- `avcodec_register_all()`：註冊所有的decoder。
- `AVCodec* avcodec_find_decoder(enum AVCodecID)`：用codec ID來找到相對應的codec instance
- `AVCodec* avcodec_find_decoder_by_name(const char* name)`：用codec的名子來找到相對應的codec instance
- `AVCodecContext* avcodec_alloc_context3(const AVCodec*)`：用codec來建立`AVCodecContext`，codec只是用來解碼，解碼之後的相關資料要放在`AVCodecContext`裡面。
- `void avcodec_free_context(AVCodecContext**)`：釋放`AVCodecContext`。
- `int avcodec_open2(AVCodecContext*, const AVCodec*, AVDictionary*)`
    - 開始解碼
    - `AVDictionary`的選項請看`libavcodec/options_table.h`

#### AVFrame
##### 重要的member
- `uint8_t* data[AV_NUM_DATA_POINTERS]`：存放這個frame的資料
- `int linesize[AV_NUM_DATA_POINTERS]`：存放指向各個plane的pointer
    - Plane mode
        - For video
            - 0：Y
            - 1：U
            - 2：V
            - 3：NULL
        - For audio
            - 0：Left channel
            - 1：Right channel
            - 2：NULL
    - Non plane mode
        - For video
            - 0：RGB RGB RGB
            - 1：NULL
- `int width`：For video, frame width
- `int height`：For video, frame height
- `int nb_samples`：For audio
- `int64_t pts`：這個frame的pts
- `int64_t pkt_dts`：Packet的dts
- `int sample_rate`：For audio，sampling rate
- `uint64_t channel_layout`：For audio
- `int channels`：For audio
- `int format`：For video it's AVPixelFormat, for audio it's AVSampleFormat

##### 操作function
- `AVFrame* av_frame_alloc()`：
- `void av_frame_free(AVFrame*)`：
- `AVFrame* av_frame_clone(const AVFrame*)`：
- `int av_frame_ref(AVFrame* dst, const AVFrame* src)`：
- `void av_frame_unref(AVFrame*)`：
- `int avcodec_send_packet(AVCodecContext*, const AVPacket*)`
- `int avcodec_receive_frame(AVCodecContext*, AVFrame*)`

## Frame scaling
##### 操作function
- `sws_getContext`
- `sws_getCachedContext`
    ```
    SwsContext* sws_getCachedContext(
        SwsContext*, 
        int srcW, int srcH, enum AVPixelFormat srcFormat,
        int dstW, int dstH, enum AVPixelFormat dstFormat,
        int flags,
        SwsFilter srcFilter,
        SwsFilter dstFilter,
        const double* param)
    ```
- `sws_scale`
    ```
    int sws_scale(
        SwsContext*,
        const uint8_t* const srcSlice[], // 來源buffer的pointer, 如果是plane mode，要丟srcSlice[3]
        const int srcStride[],           // linesize
        int srcSliceY,
        int srcSliceH,                   // image height
        uint8_t* const dst[],            // 目標buffer的pointer, 如果是plane mode，要丟dst[3]
        const int dstStride[])           // linesize
    ```
- `void sws_freeContext(SwsContext*)`

## Audio resampling
- `SwrContext* swr_alloc()`
- `swr_alloc_set_opts`
    ```cpp
    SwrContext* swr_alloc_set_opts(
        SwrContext*,
        int64_t out_ch_layout,           // Output
        AVSampleFormat outSampleFormat,  // Output
        int outSampleRate,               // Output
        int64_t in_ch_layout,            // Input
        AVSampleFormat inSampleFormat,   // Input
        int inSampleRate,                // Input
        int log_offset,                  // 一般丟0
        void* log_ctx)                   // 一般丟0
    ```
- `int swr_init(SwrContext*)`
- `void swr_free(SwrContext**)`
- `swr_convert`
    ```cpp
    int swt_convert(
        SwrContext*,
        uint8_t** outData,
        int outCount,        // Sample count for one channel
        uint8_t** inData,
        int inCount)         // Sample count for one channel
    ```
    
## Audio 播放
### 會用到的class
#### QAudioFormat 
- `setSampleRate`
- `setSampleSize`
- `setChannelCount`
- `setCodec("audio/pcm")`
- `setByteOrder(QAudioFormat::LittleEndian)`
- `setSampleType(QAudioFormat::UnSignedInt)`

#### QAudioOutput
- Constuctor參數就是一個`QAudioFormat`
- `QIODevice* start()`
- `suspend()`
- `resume()`
- `bufferSize()`
- `byteFree()`
- `periodSize()`

#### QIODevice
- `qint64 write(const char* data, qint64 len)`

#### QBuffer
```cpp
QBuffer buffer;

if (buffer.open(QIODevice::ReadWrite)) {
    qInfo() << "Buffer Opened!";
    QByteArray data("Hello World");

    for (int i = 0; i < 5; i++) {
        buffer.write(data);
        buffer.write("\r\n");
    }
    //File and device access you may need to flush the data to the device
    //buffer.flush()

    //Move to the first position
    buffer.seek(0);

    qInfo() << buffer.readAll();     // retuen a QVariant
    qInfo() << "closing the buffer";

    //ALWAYS close your device!
    buffer.close();
}
```

### Example
```cpp
QAudioFormat audioFormat;
audioFormat.setSampleRate(44100);
audioFormat.setSampleSize(16);
audioFormat.setChannelCount(2);
audioFormat.setCodec("audio/pcm");
audioFormat.setByteOrder(QAudioFormat::LittleEndian);
audioFormat.setSampleType(QAudioFormat::UnSignedInt);

QAudioOutput* audioOutput = new QAudioOutput(audioFormat);
QIODevice* ioDevice = audioOutput->start();

int size = audioOutput->periodSize();
char* buffer = new char[size];
FILE *pFile = fopen("E:\\download\\out.pcm", "rb");
while (!feof(pFile)) {
    if (audioOutput->bytesFree() < size) {
        QThread::msleep(1);
    }

    qint64 len = fread(buffer, 1, size, pFile);
    if (len <= 0) {
        break;
    }
    ioDevice->write(buffer, len);
}
fclose(pFile);
delete buffer;
buffer = NULL;
```

### 使用FFmpeg decode audio
- [音视频开发之旅（60) -调试分析FFmpeg (解封装部分的）常用结构体](https://zhuanlan.zhihu.com/p/441055685)

## 參考資料
- [mpv.io](https://mpv.io/)
- [《FFmpeg基础知识》](https://opensourcelibs.com/lib/zhangfangtaozft-ffmpeg)
- [ffmpeg - 专题 - 简书](https://www.jianshu.com/c/21a6fafc8ee3)