# アーキテクチャ構成図

## 1. システム全体構成図

```mermaid
graph TB
    subgraph Client["Client"]
        CLI[CLI / REST Client]
        WebUI[Web UI]
    end

    subgraph FlinkCluster["Flink Cluster"]
        subgraph JobManager["JobManager (Master)"]
            Dispatcher[Dispatcher]
            RM[ResourceManager]
            JMR[JobManagerRunner]
            JM[JobMaster]
            CPC[CheckpointCoordinator]
        end

        subgraph TaskManagers["TaskManagers (Workers)"]
            TM1[TaskManager 1]
            TM2[TaskManager 2]
            TM3[TaskManager N]
        end
    end

    subgraph ExternalSystems["External Systems"]
        ZK[(ZooKeeper)]
        FS[(FileSystem<br/>HDFS/S3/GCS)]
        Sources[Data Sources<br/>Kafka/Files]
        Sinks[Data Sinks<br/>Kafka/DB/Files]
    end

    CLI --> Dispatcher
    WebUI --> Dispatcher
    Dispatcher --> JMR
    JMR --> JM
    JM --> CPC
    JM --> RM
    RM --> TM1
    RM --> TM2
    RM --> TM3
    JM --> TM1
    JM --> TM2
    JM --> TM3

    JobManager --> ZK
    TM1 --> FS
    TM2 --> FS
    TM3 --> FS
    TM1 --> Sources
    TM1 --> Sinks
    TM2 --> Sources
    TM2 --> Sinks
    TM3 --> Sources
    TM3 --> Sinks
```

## 2. コンポーネント詳細図

```mermaid
graph TB
    subgraph Dispatcher["Dispatcher"]
        DispatcherMain[Dispatcher]
        JobStore[(Job Store)]
        HistoryServer[History Server]
    end

    subgraph JobMasterComponents["JobMaster Components"]
        JobMaster[JobMaster]
        Scheduler[Scheduler]
        ExecGraph[ExecutionGraph]
        SlotPool[SlotPool]
    end

    subgraph CheckpointSystem["Checkpoint System"]
        CheckpointCoord[CheckpointCoordinator]
        CheckpointStore[(Checkpoint Storage)]
        StateBackend[StateBackend]
    end

    subgraph ResourceManagement["Resource Management"]
        ResourceMgr[ResourceManager]
        SlotManager[SlotManager]
    end

    subgraph TaskExecution["Task Execution"]
        TaskExecutor[TaskExecutor]
        TaskSlot1[Slot 1]
        TaskSlot2[Slot 2]
        Task1[Task]
        Task2[Task]
        MemoryManager[MemoryManager]
        NetworkStack[NetworkStack]
    end

    DispatcherMain --> JobMaster
    JobMaster --> Scheduler
    Scheduler --> ExecGraph
    JobMaster --> SlotPool
    JobMaster --> CheckpointCoord
    CheckpointCoord --> CheckpointStore
    CheckpointCoord --> StateBackend

    SlotPool --> ResourceMgr
    ResourceMgr --> SlotManager
    SlotManager --> TaskExecutor

    TaskExecutor --> TaskSlot1
    TaskExecutor --> TaskSlot2
    TaskSlot1 --> Task1
    TaskSlot2 --> Task2
    TaskExecutor --> MemoryManager
    TaskExecutor --> NetworkStack
```

## 3. レイヤー構成図

```mermaid
graph TB
    subgraph APILayer["API Layer"]
        DataStreamAPI[DataStream API]
        TableAPI[Table API]
        SQLAPI[SQL API]
    end

    subgraph GraphLayer["Graph Translation Layer"]
        StreamGraph[StreamGraph]
        JobGraph[JobGraph]
        ExecutionGraph[ExecutionGraph]
    end

    subgraph RuntimeLayer["Runtime Layer"]
        JobMaster2[JobMaster]
        TaskExecutor2[TaskExecutor]
        NetworkShuffle[Network Shuffle]
    end

    subgraph StateLayer["State & Checkpoint Layer"]
        StateBackend2[StateBackend]
        CheckpointStorage2[CheckpointStorage]
        Savepoint[Savepoint Manager]
    end

    subgraph ClusterLayer["Cluster Management Layer"]
        ResourceManager2[ResourceManager]
        HAServices[HA Services]
        ClusterDeployment[Cluster Deployment<br/>YARN/K8s/Standalone]
    end

    DataStreamAPI --> StreamGraph
    TableAPI --> StreamGraph
    SQLAPI --> TableAPI
    StreamGraph --> JobGraph
    JobGraph --> ExecutionGraph

    ExecutionGraph --> JobMaster2
    JobMaster2 --> TaskExecutor2
    TaskExecutor2 --> NetworkShuffle

    TaskExecutor2 --> StateBackend2
    StateBackend2 --> CheckpointStorage2
    JobMaster2 --> Savepoint

    JobMaster2 --> ResourceManager2
    ResourceManager2 --> ClusterDeployment
    JobMaster2 --> HAServices
    ResourceManager2 --> HAServices
```

## 4. データフロー図

```mermaid
sequenceDiagram
    participant Client
    participant Dispatcher
    participant JobMaster
    participant ResourceManager
    participant TaskExecutor
    participant Source
    participant Sink
    participant CheckpointStorage

    Client->>Dispatcher: Submit JobGraph
    Dispatcher->>JobMaster: Create JobMaster
    JobMaster->>JobMaster: Build ExecutionGraph
    JobMaster->>ResourceManager: Request Slots
    ResourceManager->>TaskExecutor: Allocate Slots
    TaskExecutor->>JobMaster: Offer Slots
    JobMaster->>TaskExecutor: Deploy Tasks

    loop Data Processing
        Source->>TaskExecutor: Read Data
        TaskExecutor->>TaskExecutor: Process Data
        TaskExecutor->>Sink: Write Data
    end

    JobMaster->>TaskExecutor: Trigger Checkpoint
    TaskExecutor->>CheckpointStorage: Save State
    TaskExecutor->>JobMaster: Acknowledge Checkpoint
```

## 5. チェックポイント処理フロー

```mermaid
sequenceDiagram
    participant CC as CheckpointCoordinator
    participant Source as SourceTask
    participant Op1 as Operator1
    participant Op2 as Operator2
    participant Sink as SinkTask
    participant Storage as CheckpointStorage

    CC->>Source: Inject Barrier (CP-N)
    Source->>Source: Snapshot State
    Source->>Storage: Write State
    Source->>Op1: Forward Barrier

    Op1->>Op1: Wait for all barriers (align)
    Op1->>Op1: Snapshot State
    Op1->>Storage: Write State
    Op1->>Op2: Forward Barrier

    Op2->>Op2: Snapshot State
    Op2->>Storage: Write State
    Op2->>Sink: Forward Barrier

    Sink->>Sink: Snapshot State
    Sink->>Storage: Write State

    Source->>CC: Ack Checkpoint
    Op1->>CC: Ack Checkpoint
    Op2->>CC: Ack Checkpoint
    Sink->>CC: Ack Checkpoint

    CC->>CC: All Tasks Acknowledged
    CC->>Storage: Finalize Checkpoint Metadata
```

## 6. 状態バックエンド構成図

```mermaid
graph TB
    subgraph Application["Application"]
        KeyedState[Keyed State]
        OperatorState[Operator State]
        BroadcastState[Broadcast State]
    end

    subgraph StateBackends["State Backends"]
        HashMapBackend[HashMapStateBackend<br/>Heap Memory]
        RocksDBBackend[RocksDBStateBackend<br/>Local Disk + Heap]
        ForStBackend[ForStStateBackend<br/>Experimental]
    end

    subgraph CheckpointStorages["Checkpoint Storage"]
        JobManagerStorage[JobManagerCheckpointStorage<br/>JVM Heap]
        FileSystemStorage[FileSystemCheckpointStorage<br/>HDFS/S3/GCS]
    end

    KeyedState --> HashMapBackend
    KeyedState --> RocksDBBackend
    KeyedState --> ForStBackend
    OperatorState --> HashMapBackend
    OperatorState --> RocksDBBackend
    BroadcastState --> HashMapBackend
    BroadcastState --> RocksDBBackend

    HashMapBackend --> JobManagerStorage
    HashMapBackend --> FileSystemStorage
    RocksDBBackend --> FileSystemStorage
    ForStBackend --> FileSystemStorage
```

## 7. クラスタデプロイメント構成図

```mermaid
graph TB
    subgraph Standalone["Standalone Mode"]
        SM_JM[JobManager Process]
        SM_TM1[TaskManager Process 1]
        SM_TM2[TaskManager Process N]
    end

    subgraph YARN["YARN Mode"]
        YARN_RM[YARN ResourceManager]
        YARN_AM[Flink ApplicationMaster<br/>JobManager]
        YARN_C1[YARN Container<br/>TaskManager 1]
        YARN_C2[YARN Container<br/>TaskManager N]
    end

    subgraph Kubernetes["Kubernetes Mode"]
        K8s_API[Kubernetes API Server]
        K8s_JM[JobManager Pod]
        K8s_TM1[TaskManager Pod 1]
        K8s_TM2[TaskManager Pod N]
        K8s_Svc[Service/Ingress]
    end

    SM_JM --> SM_TM1
    SM_JM --> SM_TM2

    YARN_RM --> YARN_AM
    YARN_AM --> YARN_C1
    YARN_AM --> YARN_C2

    K8s_API --> K8s_JM
    K8s_API --> K8s_TM1
    K8s_API --> K8s_TM2
    K8s_Svc --> K8s_JM
```

## 8. RPC通信構成図

```mermaid
graph TB
    subgraph RPCLayer["RPC Framework"]
        RpcService[RpcService Interface]
        AkkaRpc[Akka RPC Implementation]
        RpcEndpoint[RpcEndpoint]
        RpcGateway[RpcGateway]
    end

    subgraph Endpoints["RPC Endpoints"]
        DispatcherEndpoint[DispatcherEndpoint]
        ResourceManagerEndpoint[ResourceManagerEndpoint]
        JobMasterEndpoint[JobMasterEndpoint]
        TaskExecutorEndpoint[TaskExecutorEndpoint]
    end

    subgraph Communication["Communication Patterns"]
        Request[Request/Response]
        Heartbeat[Heartbeat]
        Event[Event Notification]
    end

    RpcService --> AkkaRpc
    RpcEndpoint --> RpcGateway
    AkkaRpc --> RpcEndpoint

    DispatcherEndpoint --> Request
    ResourceManagerEndpoint --> Heartbeat
    JobMasterEndpoint --> Event
    TaskExecutorEndpoint --> Heartbeat
```

## 9. メトリクス構成図

```mermaid
graph TB
    subgraph Sources["Metric Sources"]
        JobMetrics[Job Metrics]
        TaskMetrics[Task Metrics]
        OperatorMetrics[Operator Metrics]
        SystemMetrics[System Metrics<br/>JVM/Network]
    end

    subgraph MetricSystem["Metric System"]
        MetricRegistry[MetricRegistry]
        MetricGroup[MetricGroup Hierarchy]
    end

    subgraph Reporters["Metric Reporters"]
        JMX[JMX Reporter]
        Prometheus[Prometheus Reporter]
        Datadog[Datadog Reporter]
        InfluxDB[InfluxDB Reporter]
        Graphite[Graphite Reporter]
        Slf4j[Slf4j Reporter]
    end

    subgraph External["External Systems"]
        PrometheusServer[Prometheus Server]
        DatadogCloud[Datadog Cloud]
        InfluxDBServer[InfluxDB Server]
        GraphiteServer[Graphite Server]
    end

    JobMetrics --> MetricRegistry
    TaskMetrics --> MetricRegistry
    OperatorMetrics --> MetricRegistry
    SystemMetrics --> MetricRegistry
    MetricRegistry --> MetricGroup

    MetricGroup --> JMX
    MetricGroup --> Prometheus
    MetricGroup --> Datadog
    MetricGroup --> InfluxDB
    MetricGroup --> Graphite
    MetricGroup --> Slf4j

    Prometheus --> PrometheusServer
    Datadog --> DatadogCloud
    InfluxDB --> InfluxDBServer
    Graphite --> GraphiteServer
```

## 10. モジュール依存関係図

```mermaid
graph TB
    subgraph API["API Modules"]
        CoreAPI[flink-core-api]
        Core[flink-core]
        DataStreamAPI2[flink-datastream-api]
        TableAPIJava[flink-table-api-java]
    end

    subgraph Runtime["Runtime Modules"]
        Runtime2[flink-runtime]
        StreamingJava[flink-streaming-java]
        DataStream2[flink-datastream]
        TablePlanner[flink-table-planner]
    end

    subgraph Infrastructure["Infrastructure Modules"]
        RPC[flink-rpc-core]
        RPCAkka[flink-rpc-akka]
        HA[flink-ha]
    end

    subgraph Storage["Storage Modules"]
        StateBackends2[flink-state-backends]
        FileSystems2[flink-filesystems]
    end

    subgraph Cluster["Cluster Modules"]
        YARN2[flink-yarn]
        K8s2[flink-kubernetes]
        Clients[flink-clients]
    end

    CoreAPI --> Core
    DataStreamAPI2 --> Core
    TableAPIJava --> Core

    Runtime2 --> Core
    Runtime2 --> RPC
    StreamingJava --> Runtime2
    StreamingJava --> DataStreamAPI2
    DataStream2 --> StreamingJava
    TablePlanner --> TableAPIJava
    TablePlanner --> Runtime2

    Runtime2 --> StateBackends2
    Runtime2 --> FileSystems2

    YARN2 --> Runtime2
    K8s2 --> Runtime2
    Clients --> Runtime2
```
