Fourier Analysis Networks (FAN) Implementation

This repository contains an implementation of Fourier Analysis Networks (FAN) as described in the paper "FAN: Fourier Analysis Networks" by Yihong Dong, Ge Li, et al

Table of Contents

• Introduction
• Requirements
• Usage
• Implementation Details
• Experiments
• Contributing
• Citation
• License

Introduction

Fourier Analysis Networks (FAN) is a novel neural network architecture designed to efficiently model and reason about periodic phenomena. By introducing Fourier Series into the structure and computational processes of neural networks, FAN achieves more accurate expression and prediction of periodic patterns.

Key features of FAN:

• Enhanced ability to model periodicity
• Fewer parameters and FLOPs compared to traditional MLPs
• Improved performance on various tasks, including symbolic formula representation, time series forecasting, and language modeling

Requirements

Just PyTorch...

Usage

Here's a basic example of how to use the FAN layer:

import torch
from fan import FANLayer

# Initialize a FAN layer
input_dim = 64
output_dim = 128
fan_layer = FANLayer(input_dim, output_dim)

# Create a random input tensor
x = torch.randn(32, input_dim)  # batch size of 32

# Pass the input through the FAN layer
output = fan_layer(x)

print(output.shape)  # Should be torch.Size([32, 128])

Implementation Details

The core of this implementation is the FANLayer class. This layer implements the FAN architecture as described in the paper, including:

• Initialization of learnable parameters W_p, W_p̄, and B_p̄
• Forward pass computation of cos(W_p x), sin(W_p x), and σ(B_p̄ + W_p̄ x)
• Concatenation of the computed terms

The implementation also includes variants such as FAN (Gated) and models that combine FAN with other architectures like Transformer.

Experiments

Soon we will test FAN on downstream tasks.

Contributing

We welcome contributions to improve the implementation or add new features. Please feel free to submit pull requests or open issues for any bugs or feature requests.

Citation

If you use this implementation in your research, please cite the original paper:

@article{dong2024fan,
  title={FAN: Fourier Analysis Networks},
  author={Yihong Dong and Ge Li and Yongding Tao and Xue Jiang and Kechi Zhang and Jia Li and Jing Su and Jun Zhang and Jingjing Xu},
  journal={arXiv preprint arXiv:2410.02675},
  year={2024}
}

License

This project is licensed under the MIT License - see the LICENSE file for details.