1. 6T SRAM Cell Design

Fig.1 : SRAM Cell

This shows a standard 6T SRAM cell composed of six transistors: two cross-coupled CMOS inverters for data storage and two access transistors controlled by the wordline (WL). The cell stores one bit of data by maintaining complementary internal nodes (Q and QB), while the bitlines (BL and BLB) are used for read and write operations. This structure provides fast access, low static power consumption, and robust data retention, making it the most widely used SRAM cell architecture.

Schematic

Fig.2 : SRAM Cell Schematic

Based on the standard 6T SRAM architecture, the schematic was implemented using two cross-coupled CMOS inverters and two access transistors. The cell stores data at nodes Q and QB, while BL, BLB, and WL are used for read and write operations.

Functionality Verification

Fig.3 : Read Verification

The read operation was successfully verified, as the stored data remained unchanged during the read access while WL was asserted and both bitlines were precharged to VDD.

Fig.4 : Write Verification

The write operation was successfully verified, as the stored data changed from (Q, QB) = (1, 0) to (0, 1) when WL was asserted with BL = 0 and BLB = 1.

Static Noise Margin

Fig.5 : Static Noise Margin

The hold static noise margin (SNM) of the designed 6T SRAM cell was measured to be approximately 282 mV from the butterfly curve, indicating stable data retention in the hold state.

2. SRAM Column Design

4-to-16 Decoder with Enable Signal

The 4-to-16 decoder is used as the row decoder in the 16×16 SRAM array. Since the array contains 16 rows, four address bits are required to uniquely select one row. The decoder converts the 4-bit row address into one of sixteen wordline signals (WL0–WL15), ensuring that only the selected row is activated during read or write operations. Without the decoder, it would not be possible to access an individual row within the SRAM array. Therefore, the 4-to-16 decoder is an essential component that enables correct row selection and memory access in the 16×16 SRAM design.

Fig.6 : 2:4 Decoder

As a building block for the 4-to-16 decoder, a 2-to-4 decoder was first designed and implemented. The schematic is shown above.

Fig.7 : 4:16 Decoder

To implement the 4-to-16 decoder, a hierarchical structure based on five 2-to-4 decoders was used. As shown in the schematic, the upper 2-to-4 decoder receives the two most significant address bits (A3 and A2) and generates four enable signals (B0–B3). These enable signals are connected to the enable inputs of four lower 2-to-4 decoders. Each lower decoder receives the two least significant address bits (A1 and A0) and generates four wordline outputs. As a result, only one of the sixteen outputs (WL0–WL15) is asserted for a given 4-bit address, realizing a complete 4-to-16 decoding function.

Bit Line Sense Amplifier (BLSA)

Fig.8 : BLSA

The Bit-Line Sense Amplifier (BLSA) detects a small voltage difference between INP and INN and amplifies it to full-swing digital outputs. When SAEN is low, the sensing circuit is disabled. When SAEN is asserted, the differential input pair converts the input voltage difference into a current difference. The cross-coupled latch then provides positive feedback, rapidly amplifying the small input difference until one output is driven to logic high and the other to logic low. As a result, even a 100 mV input difference can be reliably sensed and converted into complementary outputs (OUTP and OUTN). In an SRAM array, the BLSA is essential because the read operation produces only a small voltage difference on the bitlines, which must be amplified into full-swing digital signals for reliable data sensing.

Fig.9 : BLSA Schematic

The BLSA schematic was designed as shown above for reliable sensing of bitline voltage differences in the SRAM array.

Write Driver and Precharge

Fig.10 : Write Driver

The write driver is a circuit that drives the bitlines (BL and BLB) with complementary data values during a write operation. As shown above, when the Write signal is asserted, the write driver forces BL and BLB to opposite logic levels according to the input DATA, allowing the selected SRAM cell to be overwritten with the new value.

In an SRAM array, the write driver is essential because the cross-coupled inverters inside the SRAM cell are designed to strongly hold their stored data. Therefore, the bitlines must be actively driven to the desired logic levels with sufficient strength to overcome the stored value and successfully write new data into the selected cell. Without a write driver, reliable write operations would not be possible.

Fig.11 : Write Driver Schematic

Instead of using a single NMOS pass transistor, transmission gates were employed in the write driver to achieve full voltage swing on the bitlines. A single NMOS suffers from a threshold-voltage drop when passing a logic high level, resulting in a degraded output voltage. In contrast, a transmission gate combines an NMOS and a PMOS, enabling both strong logic ‘1’ and strong logic ‘0’ transmission. Therefore, the proposed design can drive BL and BLB closer to the full VDD and GND levels, improving write reliability.

Fig.12 : Precharge Circuit

The precharge circuit initializes both bitlines (BL and BLB) to VDD and equalizes their voltages before a read operation. When the Precharge signal is asserted, both bitlines are charged to the same voltage level. In an SRAM array, the precharge circuit is essential because it provides a known initial condition for the bitlines, allowing the sense amplifier to accurately detect the small voltage difference generated during a read operation.

Fig.13 : Precharge Schematic

The precharge circuit shown above was designed to precharge and equalize BL and BLB before each read operation, enabling reliable sensing in the SRAM array.

1 Column with 16bit SRAM cell

Fig.14 : SRAM Column

As shown in the figure, a single SRAM column consists of several key components:

4-to-16 Decoder: Selects one of the sixteen wordlines (WLs) based on the input address. SRAM Cells: Sixteen 6T SRAM cells are connected vertically in the column. Each cell shares the same bitline pair (BL and BLB) but has a different wordline.

Precharge Circuit: Precharges and equalizes BL and BLB before every read operation.

Write Driver: Drives BL and BLB with complementary data values during write operations. Bit-Line Sense Amplifier (BLSA): Detects and amplifies the small voltage difference developed on BL and BLB during read operations.

During operation, the decoder activates a single wordline, allowing one SRAM cell to communicate with the shared bitlines. The write driver is used for write operations, while the BLSA is used for read operations.

Fig.15 : SRAM Column Schematic

The SRAM column circuit shown above was designed.

Functionality Verification

Write functionalify verification (write 0→1→0→1 .. )

Fig.19-21 : SRAM Column WRITE

The column write operation was verified using the three transient simulation results shown above.

In the first waveform, the outputs of the 4-to-16 decoder (WL0–WL15) are monitored. As the address inputs A3, A2, A1, A0 change sequentially, only one wordline is asserted at a time, confirming correct row selection during the write operation.

The second waveform shows the internal storage nodes (Q and QB) of the SRAM cells in the selected column. Whenever a wordline is activated and the write driver applies the input data to BL and BLB, the corresponding SRAM cell changes its stored value. The complementary behavior of Q and QB confirms that the selected cell is successfully written while the unselected cells retain their previous states.

The third waveform shows the address signals, DATA input, control signals, bitline voltages (BL and BLB), and sense amplifier outputs. During each write cycle, BL and BLB are driven to complementary values according to DATA by the write driver. The selected SRAM cell captures the new value through the activated wordline. The bitline transitions match the intended DATA pattern, demonstrating that the write driver, decoder, and SRAM cell operate correctly together.

Therefore, the three simulation results verify that the column can correctly select a target row, drive the bitlines with the desired data, and successfully store the new value in the selected SRAM cell during write operations.

Write functionalify verification (write 1→0→1→0 .. )

Fig.16-18 : SRAM Column WRITE

The write operation was verified using the three waveforms shown above. The decoder waveform confirms that only one wordline (WL0–WL15) is activated at a time according to the address inputs. The BL and BLB waveforms show complementary bitline voltages driven by the write driver based on the DATA input. Finally, the Q and QB waveforms demonstrate that the selected SRAM cell successfully stores the written value while maintaining complementary logic states. These results confirm correct operation of the decoder, write driver, and SRAM cells during column write operations.

Read Functionality Verification

Fig.22-23 : SRAM Column READ

The read operation was verified using the waveforms shown above. The decoder outputs confirm that only one wordline (WL0–WL15) is activated at a time according to the address inputs. During each read cycle, the selected SRAM cell creates a small voltage difference between BL and BLB after precharge. When SAEN is asserted, the BLSA senses this differential voltage and amplifies it to full-swing outputs (OUTP and OUTN). The output waveforms correctly correspond to the data stored in the selected SRAM cells, confirming successful read operation of the SRAM array.

3. 16x16 SRAM Array Design

Fig.24 : SRAM Array

This figure illustrates the overall architecture of the 16×16 SRAM array. A single 4-to-16 decoder is shared across all columns to generate the wordlines, while each column contains 16 SRAM cells connected to a dedicated precharge circuit, write driver, and BLSA. The array was designed by replicating the single-column SRAM structure sixteen times, enabling storage and access of 16-bit words in parallel.

Schematic

Fig.25 : SRAM Array Schematic

The complete 16×16 SRAM array schematic was implemented as shown above. A single 4-to-16 decoder generates the wordline signals shared by all columns, while sixteen identical SRAM columns are instantiated in parallel. Each column contains a 16-cell SRAM array, a precharge circuit, a write driver, and a BLSA, enabling parallel storage and access of 16-bit data.

Functionality Verification

Fig.26-28 : SRAM Array READ & WRITE Verification

To verify the functionality of the proposed 16×16 SRAM array, two rows, WL6 and WL7, were selected for write and read operations. First, the data pattern 1010101010101010 (0xAAAA) was written into WL6, followed by the complementary pattern 0101010101010101 (0x5555) written into WL7. During the write cycles, the selected wordline was activated and the write driver forced the bitlines according to the input data, causing the corresponding SRAM cells to update their stored values. As a result, WL6 and WL7 successfully stored different 16-bit data patterns.

For the read operation, the bitlines were precharged to VDD, after which the selected wordline generated a small voltage difference between BL and BLB depending on the stored data. When the SAEN signal was asserted, the BLSA amplified this differential voltage and produced full-swing output signals. The read results matched the data previously written into WL6 and WL7, confirming successful data storage and retrieval. These results verify the correct operation of the decoder, write driver, precharge circuit, SRAM cell array, and BLSA as an integrated 16×16 SRAM array.

4. Layout

SRAM Cell

Fig.29 : SRAM Cell Layout

The designed SRAM cell occupies an area of 2.05μm×1.95μm, resulting in a total cell area of 3.9975μm2.

Body Tap Cell

Fig.30 : Body Tap Cell Layout

This figure shows the layout of the body tap cell used in the SRAM array. The body tap cell provides a direct connection between the transistor bodies and the power rails (VDD and VSS), ensuring proper body biasing and preventing latch-up. In the overall layout, body tap cells are inserted periodically to maintain substrate and well potentials.

4-to-16 Decoder

Fig.31 : 2-to-4 decoder Layout

This figure shows the layout of 2-to-4 decoder.

Fig.32 : 4-to-16 decoder Layout

4-to-16 decoder was designed, based on the 2-to-4 decoder layout.

BLSA

Fig.33 : BLSA Layout

This figure shows the layout design of BLSA. The width of the designed layout is identical to that of SRAM cell.

Write Driver

Fig.34 : Write Driver Layout

This figure shows the layout design of write driver. The width of the designed layout is identical to that of SRAM cell.

Precharge

Fig.35 : Precharge Layout

This figure shows the layout design of precharge circuit. The width of the designed layout is identical to that of SRAM cell.

1 column with 16bit SRAM

Fig.36 : 1 Column with 16bit SRAM Layout

This figure shows the layout design of SRAM column. For one column, the body tap cells were inserted at the top, in the middle, and at the bottom to prevent latch-up.

Fig.37 : 1 Column with Peripheral Circuits Layout

This figure shows the layout design of SRAM 1 column, with peripheral circuits such as precharge, BLSA and write driver.

16x16 SRAM Array

Fig.38 : SRAM array Layout

The final layout implementation of the 16×16 SRAM array, including the decoder, SRAM cells, precharge circuits, write drivers, BLSAs, and body tap cells, is shown above.

1. Introduction to Sklansky Adder

PGD Notation

Fig.1 : PGD Notation

PGD (Propagate, Generate, Delete) notation is a method used to describe carry behavior in a full adder based only on the input bits A and B.

Generate (\(\text{G} = \text{A} \cdot \text{B}\)): The adder generates a carry regardless of the input carry, so \(\text{C}_{out}=1\). Propagate (\(\text{P} = \text{A} \oplus \text{B}\)): The input carry is propagated to the output, so \(\text{C}_{out}=\text{C}_{in}\).

Delete (\(\text{D} = \overline{A}\cdot\overline{B}\)): The carry is deleted regardless of the input carry, so \(\text{C}_{out}=0\).

Using PGD notation, the carry and sum outputs can be expressed as:

This notation simplifies carry analysis and is widely used in high-speed adder designs such as carry-lookahead and parallel-prefix adders.

Look Ahaed Adder

Fig.2 : Look Ahaed Adder

The basic idea of carry lookahead logic. Instead of waiting for the carry signal to ripple through each bit one by one, the circuit computes all carry signals \(\text{C}_{i}\) in parallel using the propagate (P) and generate (G) signals. Once the carry for each bit is available, the sum can be calculated as

\(\text{S}_{i} = \text{P}_{i} \oplus \text{C}_{i} = \text{A}_{i}\oplus \text{B}_{i} \oplus\text{C}_{i}\).

The carry/propagation computation block receives all input bits (\(\text{A}_{i}, \text{B}_{i}\)) and quickly determines the carry for every stage. This parallel carry computation significantly reduces delay compared to a ripple-carry adder, making carry lookahead adders much faster for large bit-width additions.

Black Cell, Gray Cell and Buffer

Fig.3 : Black cell, Gray cell and Buffer

There are fundamental building blocks used in parallel-prefix adders, such as the Kogge–Stone adder. These cells combine propagate (P) and generate (G) signals to efficiently compute carry information across multiple bit positions.

Black Cell: Combines two PG pairs and produces both a new propagate and a new generate signal:

Since it outputs both P and G, it is used in intermediate stages of the prefix network.

Gray Cell: Computes only the generate signal:

Because only the carry information is needed at the final stage, the propagate output is omitted, reducing hardware complexity.

Buffer: Simply forwards the existing P and G signals without modification. It is used to balance timing and maintain a regular network structure.

By repeatedly combining PG signals with black and gray cells, a parallel-prefix adder can calculate carries for all bits in parallel, achieving much lower delay than a ripple-carry adder.

Sklansky Adder

Fig.4 : Sklansky Adder

The structure of a 16-bit Sklansky parallel-prefix adder is as the above image, which is designed to compute carry signals with low logic depth. The network combines propagate (P) and generate (G) signals through a hierarchy of black and gray cells to determine carries for all bit positions.

A key advantage of the Sklansky adder is that it requires only \(\log_2(n)\) stages, allowing carry signals to be generated much faster than in a ripple-carry adder. At each stage, larger groups of bits are combined, such as (10:8) and (7:0), to form wider carry prefixes.

However, the Sklansky structure has some drawbacks. The distribution of logic is uneven, meaning that some nodes drive significantly more outputs than others. This results in large fanout, especially near the most significant bits, which can increase delay and power consumption in practical implementations. Despite this limitation, the Sklansky adder remains attractive because of its small logic depth and relatively low hardware complexity.

2. Basic Cell Design for Carry Lookahead Adder

Setup Cell

Fig.5 : Setup Cell

Carry Cell

Fig.6 : Carry Cell

Sum Cell

Fig.7 : Sum Cell

Black Cell

Fig.8 : Black Cell

Gray Cell

Fig.9 : Gray Cell

3. 16bit Carry Lookahead Adder Design

Sklansky Adder

Fig.10 : Sklansky Adder

Sklansky Adder with Flip Flops

Fig.11 : Sklansky Adder with Flip Flops

Fig.12 : Sklansky Adder with Flip Flops Design

This figure shows the application circuit used to verify the 16-bit Sklansky adder. To accurately evaluate the adder under realistic operating conditions, flip-flops (FFs) were placed at both the input and output sides of the adder.

The input flip-flops register the 16-bit operands A[15:0] and B[15:0] on the rising edge of the clock and provide stable inputs to the Sklansky adder. The adder then performs the addition and generates the sum S[15:0] and carry-out (Cout). The output flip-flops capture these results at the next clock edge, enabling synchronous operation and accurate timing measurements.

By surrounding the combinational adder with input and output registers, the circuit can be simulated and analyzed as a pipelined system, allowing the propagation delay and overall performance of the 16-bit Sklansky adder to be verified more reliably.

Functionality Verification

Case 1.

Fig. 13-14 : Case 1

A=0000_0000_0000_0000 → 0000_1111_0000_1111

B=0000_0000_0000_0000 → 1111_0000_1111_0000

S=0000_0000_0000_0000 → 1111_1111_1111_1111

Cout = 0

Case 2.

Fig. 15-16 : Case 2

A=1111_1111_1111_1111

B=0000_0000_0000_0000 → 0000_0000_0000_0001

S=1111_1111_1111_1111 → 0000_0000_0000_0000

Cout = 0 → 1

Case 3.

Fig. 17-18 : Case 3

A = 0000 0000 0000 0000 → 0000 0000 0000 0001

B = 0000 0000 0000 0000 → 1111 1111 1111 1111

S = 0000 0000 0000 0000 → 0000 0000 0000 0000

Cout = 0 → 1

Although the final sum output is zero, temporary glitches were observed on the sum bits during the input transition. This occurs because the propagate signals reach the sum XOR gates earlier than the carry signals generated through the prefix tree. As a result, the sum output temporarily changes before the correct carry arrives, and then settles to the correct final value. Therefore, these glitches are caused by unequal path delays in the combinational Sklansky adder and do not indicate a functional error.

This case occurs the worst case delay since it is a pattern such that a carry is generated at the least significant bit and propagated through all higher bits to the final carry-out. In this case, bit 0 generates a carry, while bits 1 through 15 are in the propagate condition. Therefore, the carry must travel through the longest prefix path of the 16-bit Sklansky adder, producing the worst-case delay at Cout.

4. Layout Design of Basic cells

Setup Cell

Fig.19 : Setup Cell

Carry Cell

Fig.20 : Carry Cell

Sum Cell

Fig.21 : Sum Cell

Black Cell

Fig.22 : Black Cell

Gray Cell

Fig.23 : Gray Cell

TGFF

Fig.24-25 : TGFF Schematic and Layout

5. Layout Design of 16bit sklansky adder

Fig.26 : Floorplan

This figure illustrates the floorplan of the 16-bit Sklansky adder, where 16 one-bit adder units are arranged vertically between input and output TGFFs to achieve a regular layout structure while satisfying the 49.28 μm center-to-center spacing requirement of the Metal2 power rails.

Fig.27 : Final Layout

This figure shows the final layout implementation of the 16-bit Sklansky adder in Cadence Virtuoso, where the TGFFs are placed on both sides of the adder core and the routing is completed while maintaining the required power rail structure and layout constraints.

1. Implementation of Scaled Dot-Product Attention Using CUDA Kernels

The scaled dot-product attention operation was implemented on NVIDIA RTX 4060 Ti, RTX 5080, and L40S GPUs. Various optimization techniques previously applied to matrix multiplication on the Ada architecture were adapted to the attention computation.

The computation pipeline consisted of three kernels. The (QK^T) matrix multiplication and attention mask addition were fused into a single kernel, followed by a softmax kernel and a matrix multiplication kernel for (PV). Through this design, the entire scaled dot-product attention could be executed using only three CUDA kernels.

1.1 Shared Memory Tiling

A tiled matrix multiplication strategy was first implemented by loading partial matrices into shared memory, computing partial products, and accumulating the results.

Since the (QK^T) operation requires multiplication with a transposed (K) matrix, data were loaded from global memory in a non-transposed format to maximize memory coalescing. The transposition was then performed when storing the data into shared memory.

To further improve performance, 1D and 2D tiling techniques previously used in matrix multiplication were applied. These approaches reduce the frequency of accesses to both global memory and shared memory, thereby increasing data reuse and improving execution speed compared to a naïve shared-memory implementation.

Furthermore, although CUDA compilers do not automatically vectorize accesses to contiguous global memory locations, vectorized memory access functions were explicitly utilized, resulting in additional performance improvements.

1.2 Warp-Level Optimization Using WMMA

Profiling results revealed that global memory loading was the primary bottleneck for FP32 computation, whereas computation itself became the dominant bottleneck in FP16. Since most modern deep learning models operate in low-precision formats such as FP16 and BF16, optimization efforts focused on FP16 execution.

GPUs follow a SIMT execution model. Threads within an SM execute the same instruction stream, and instructions are scheduled at the warp level. On modern NVIDIA GPUs, a warp consists of 32 threads.

Leveraging this execution model, the matrix multiplication operations within scaled dot-product attention were scheduled at the warp level. Since warps can execute independently and in parallel, this approach increases the overall degree of parallelism.

In addition, modern GPUs contain Tensor Cores designed specifically to accelerate matrix-matrix multiplication. Tensor Cores operate at the warp level, and NVIDIA provides WMMA APIs for accessing them. These WMMA operations were integrated into the matrix multiplication stage of scaled dot-product attention.

As a result, on the RTX 5080 GPU, the optimized implementation achieved a runtime of 0.87 ms for a sequence length of 512, matching the performance of PyTorch. However, for longer sequence lengths such as 1024 and 2048, the implementation remained approximately 1–4 ms slower than the PyTorch baseline.

1.3 Instruction-Level Optimization Using PTX

Despite multiple optimization stages, the implementation remained slower than the PyTorch baseline. Further profiling revealed a significant number of shared memory bank conflicts.

The performance gap became increasingly pronounced as matrix dimensions increased, suggesting that bank conflicts were a major bottleneck.

With WMMA, thread access patterns inside a warp cannot be directly controlled, making it difficult to avoid multiple threads accessing the same shared memory bank.

To address this issue, PTX instructions provided by NVIDIA were employed. NVIDIA provides instructions that permute addresses when storing data into shared memory. By utilizing these instructions, threads within the same warp were forced to access different shared memory banks, effectively eliminating bank conflicts.

As a result, the optimized implementation outperformed the PyTorch baseline across all sequence lengths that could fit within the memory capacity of the RTX 5080 GPU, up to a sequence length of 4096.

In particular, for sequence length 4096, PyTorch required 50.18 ms, whereas the PTX-optimized custom kernel completed in 39.92 ms, achieving approximately a 10 ms speedup.

1.4 Softmax Optimization

While optimizing matrix multiplication, kernel-level profiling indicated that the softmax operation constituted a major performance bottleneck.

An online softmax algorithm was initially implemented to simultaneously compute the running maximum, perform numerical stabilization, and accumulate the normalization term. However, this approach required storing intermediate values in shared memory and introduced multiple synchronization barriers through repeated __syncthreads() calls.

To eliminate these overheads, the warp-level execution model used in matrix multiplication was applied to softmax computation as well. Each warp independently processed one softmax operation, removing the need for shared memory communication and synchronization.

Through this design, a softmax kernel with performance comparable to PyTorch’s implementation was achieved.

2. Implementation of Attention Linear Projection Using CUDA Kernels

Matrix multiplication kernels were implemented to perform the Q/K/V projections and the output linear projection in the attention mechanism.

2.1 Shared Memory Tiling

A cache-blocked tiled GEMM kernel was implemented using shared memory. Input and weight matrices were loaded into shared memory tile-by-tile, and partial products were accumulated locally.

The primary objective was to reduce redundant global memory accesses and improve arithmetic intensity through tile reuse.

During implementation, considerations such as shared memory bank conflicts and boundary handling for incomplete tiles were required.

Performance evaluation showed that the implementation remained slower than cuBLAS, indicating the need for further optimization. Profiling suggested that memory access overhead rather than computational workload was the dominant bottleneck, motivating the adoption of register tiling as the next optimization stage.

In addition, accuracy verification revealed errors significantly exceeding acceptable tolerance levels, indicating issues in the computation process that required further investigation.

2.2 Accuracy Issues

For correctness verification, the outputs of the custom kernel were compared against cuBLAS results using the maximum absolute difference metric.

It was discovered that cuBLAS assumes column-major matrix layouts by default, whereas the custom kernel interpreted matrices in row-major format. This layout mismatch effectively introduced an implicit transpose operation and resulted in substantial numerical discrepancies.

After aligning the matrix layouts, the accuracy issue was resolved. Boundary conditions were also carefully examined to ensure that no out-of-bound accesses or missing computations occurred near tile boundaries.

2.3 Register Block Tiling: 1D and 2D

Register block tiling was introduced by allowing each thread to accumulate a subset of output elements directly in registers.

Both 1D and 2D block tiling strategies were implemented sequentially, increasing per-thread workload and improving data reuse.

Compared with the shared-memory-only implementation, register tiling improved performance by approximately 40%.

While 2D tiling further increases data reuse relative to 1D tiling, its effectiveness depends on the underlying bottleneck. If shared memory bandwidth or bank conflicts dominate execution time, additional tiling may provide limited benefits. Furthermore, increased register usage can reduce achieved occupancy due to higher register pressure.

These observations highlight the importance of balancing tile dimensions, shared memory utilization, and register consumption through careful parameter tuning.

2.4 Vectorization

To improve memory efficiency, vectorized memory accesses were introduced.

Contiguous memory locations were loaded as vectorized data structures both during global-to-shared memory transfers and shared-to-register transfers. This reduced the number of memory transactions and improved memory throughput.

Performance measurements showed approximately a 30% improvement after vectorization.

Additional optimization opportunities were identified through shared memory indexing refinements and tuning of tile parameters such as BK.

3. Implementation of Q/K/V Projection Using CUDA Kernels

A GEMM kernel was implemented for generating Q, K, and V projections in the attention mechanism.

The target problem consisted of FP16 input matrices (A) and (B), FP32 accumulation matrices (C) and (D), and matrix dimensions (M = N = K = 4096). The ultimate objective was to achieve performance comparable to cuBLAS.

3.1 Naïve MMA Kernel

3.1.1 Kernel Design

The simplest approach directly invokes the PTX instruction

mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32

to execute Tensor Core operations.

On Ada GPUs, each SM contains four warp schedulers and Tensor Cores, requiring at least four warps per thread block to fully utilize the hardware.

A thread block consisting of (16 \times 16) threads was used, yielding a total of eight warps. These warps were arranged as a (2 \times 4) grid, collectively computing a (32 \times 32) output tile.

Each warp computes a (16 \times 8) output tile, and all Tensor Core operations are issued through inline PTX assembly rather than CUDA high-level APIs.

3.1.2 Data Movement and Computation

The naïve kernel employs a straightforward data movement strategy.

Each thread independently loads FP16 elements from global memory into shared memory. These accesses are neither contiguous nor aligned, resulting in poor global memory utilization.

Subsequently, data are loaded from shared memory into registers. Since multiple threads within the same warp frequently access the same shared memory bank, substantial bank conflicts occur.

Because each loaded element participates in only a single MMA operation, data reuse is extremely limited. Consequently, memory access costs dominate execution time and Tensor Core resources remain underutilized.

3.1.3 Performance Analysis

Kernel 1.0 achieved an execution time of 4680 µs and a throughput of 29.4 TFLOP/s.

This corresponds to approximately 19.1% of cuBLAS performance and 17.8% of the theoretical peak performance of the RTX 4090.

Nsight Compute analysis identified shared memory throttling, synchronization barriers, and long-scoreboard stalls caused by global memory loads as the primary bottlenecks.

On average, each MMA instruction required approximately 180 cycles, substantially higher than the theoretical minimum of 32 cycles.

3.1.4 Basic Tiling Improvement: Kernel 1.1

Applying 2× tiling along both the M and N dimensions allows each warp to execute four MMA operations per iteration.

As a result, the output tile size expands from (32 \times 32) to (64 \times 64).

This simple modification reduces execution time to 2400 µs and increases throughput to 57.3 TFLOP/s, achieving approximately 37.3% of cuBLAS performance.

However, global memory inefficiencies and shared memory bank conflicts remain unresolved, limiting further performance gains.

3.2 Vectorized Loads and Permuted Shared Memory Layout

3.2.1 Motivation

The primary bottlenecks of Kernel 1 were inefficient global memory accesses, shared memory bank conflicts, and low data reuse.

To address these issues, techniques inspired by CUTLASS were adopted, including 128-bit vectorized loads using uint4, permuted shared memory layouts, and warp-level loading via ldmatrix.

3.2.2 Vectorized Global Memory Loads

Tensor Cores operate on 128-bit vector fragments. Therefore, eight consecutive FP16 values along the K dimension were packed into a single uint4 load.

Matrices A and B were stored in row-major and column-major formats, respectively. Warp-level loading ensured fully coalesced global memory accesses and significantly improved bandwidth utilization.

3.2.3 Permuted Shared Memory Layout

Shared memory consists of 32 banks with a width of 4 bytes.

Since a uint4 occupies 16 bytes, naïve row-major storage would generate bank conflicts during warp-wide accesses.

To avoid this, an XOR-based permutation was applied:

store_column = (laneID mod 8) XOR floor(laneID / 8)

This permutation eliminates bank conflicts both during global-to-shared stores and subsequent ldmatrix loads.

3.2.4 Register Loading Using ldmatrix

The instruction ldmatrix.sync.aligned loads (8 \times 128)-bit blocks from shared memory and transforms them into Tensor Core register fragments.

A tiles are loaded using ldmatrix.x4, while B tiles are loaded using ldmatrix.x2.

Because of the permuted shared memory layout, all threads access distinct banks, eliminating loading stalls.

3.2.5 Performance Improvements

Kernel 2.0 reduced execution time to 1080 µs and achieved 127.3 TFLOP/s, corresponding to 82.9% of cuBLAS performance.

Kernel 2.1 further improved register reuse by loading A tiles only once, reducing execution time to 1030 µs and increasing throughput to 133.4 TFLOP/s, or 86.9% of cuBLAS performance.

Average MMA latency decreased to 38 cycles, and the dominant bottleneck shifted from memory stalls to Tensor Core pipeline utilization.

3.3 N-Stage Global-to-Shared Pipeline

3.3.1 Motivation

Although Kernel 2 significantly improved performance, long-scoreboard stalls caused by global memory loads remained.

To further hide memory latency, an N-stage global-to-shared pipeline based on cp.async was introduced.

3.3.2 cp.async Pipeline Design

The cp.async instruction enables asynchronous data transfers from global memory to shared memory.

A circular buffer consisting of N_STAGES shared-memory buffers was allocated, and synchronization was controlled through cp.async.commit_group and cp.async.wait_group.

During execution, MMA operations on the current stage overlap with global-to-shared prefetching for the next stage, effectively hiding memory latency.

3.3.3 Performance Results and Extended Tiling

Kernel 3.0 achieved an execution time of 1000 µs and a throughput of 137.4 TFLOP/s, corresponding to 89.5% of cuBLAS performance.

Kernel 3.1 further expanded tiling along both M and N dimensions by 4×, increasing MMA density per warp.

As a result, execution time decreased to 895 µs, matching the performance of cuBLAS.

Warp-state analysis showed that most stalls were due to Tensor Core pipeline limitations, and average MMA latency reached 34.2 cycles, approaching the theoretical minimum of 32 cycles.

3.3.4 Summary

By combining an N-stage global-to-shared pipeline with large-scale tiling, global memory latency was effectively hidden and Tensor Core utilization was maximized.

These results demonstrate that achieving near-theoretical Tensor Core performance requires careful memory-layout design, warp-level data movement optimization, and asynchronous pipeline scheduling.

References

[1] https://www.spatters.ca/mma-matmul

[2] https://siboehm.com/articles/22/CUDA-MMM

you can find the whole paper : LLM on FPGA: Squeezing Language Models by Quantization and Multi-Query Attention and its Efficient Hardware Architecture link

The paper got recognized “For enabling analytical evaluation of various quantization schemes (SmoothQuant and QLoRA) and implementing on-chip caches of large language models without siginificant accuracy drop by using Multi-Query Attention and efficient RTL design.”

LLM on FPGA: Squeezing Language Models by Quantization and Multi-Query Attention and Its Efficient Hardware Architecture

Authors:

Seoyoon Chae (Dept. of Electronic and Electrical Engineering, Sungkyunkwan University)

Taewook Kang (Dept. of Semiconductor Convergence Engineering, Sungkyunkwan University)

🔗 Code: https://github.com/chae-sy/squeezing_lm

Abstract

We present an on-chip implementation of a compressed Transformer-based language model on a Xilinx Artix-7 FPGA.

Our main contributions are:

1. Hybrid compression: Combining 4-bit quantization and multi-query attention (MQA) compresses the KV cache by 8×, supporting sequences up to 256 tokens.
2. Streaming hardware design: A Verilog implementation of pre-layernorm, attention, and feed-forward sublayers optimized with BRAM and DSP utilization.
3. Post-synthesis performance: Achieving 4.4 K tokens/s at 250 MHz with 31.9% BRAM and 85% DSP utilization.

This prototype enables fully on-chip generative inference, advancing privacy-preserving and edge-deployable large language models.

Keywords: Large Language Model (LLM), Quantization, Multi-Query Attention, Transformer Accelerator, FPGA, On-Device Inference, Hardware–Software Co-Design

1. Introduction

Transformer-based large language models (LLMs) achieve state-of-the-art performance but require substantial compute and memory resources Autoregressive generation typically caches key–value (KV) tensors, reducing attention complexity from O(S²d) to O(Sd) per token. However, storing 8-bit KV tensors on-chip severely limits sequence length.

For example, with 608 KiB BRAM on an Artix-7 FPGA, the on-chip KV capacity is:

To overcome this limitation, we integrate sub-8-bit quantization and MQA to compress the KV cache over 8×, enabling 256-token inference without DRAM.

Key contributions:

• 4-bit KV quantization using SmoothQuant + LoRA-QAT achieves near-lossless accuracy (< 0.02 drop).
• MQA (shared K/V across heads) cuts bandwidth while preserving quality.
• FPGA co-design: A streaming Transformer block implemented entirely in Verilog.

2. Related Work

Previous research explored:

• Multi-head attention acceleration on systolic arrays [2].
• ASICs using per-vector 4-bit scaling [3].
• FPGA frameworks applying pruning [4].

However, none jointly exploit sub-8-bit quantization and KV caching for generative LLM inference under FPGA-scale memory constraints.

3. Model Compression Methodology

3.1 SmoothQuant for Range Equalization

SmoothQuant [5] was used to mitigate activation outliers before quantization.

• Stable results for W8A8–W6A6 (PPL ≈ 60, Accuracy ≈ 0.6).
• Instability observed at W4A4 (PPL = 116.1, Accuracy = 0.49).

3.2 LoRA-Based Quantization-Aware Training

LoRA adapters (rank = 8) were inserted into Q/K/V projections and fine-tuned on WikiText-2 for 10–100 steps (frozen backbone).

• PPL = 45.5 after 100 steps.
• Combining with SmoothQuant reduced PPL to 22.2 (100 steps) and 55.6 (10 steps).

3.3 Multi-Query Attention (MQA)

Standard MHA computes separate K/V per head → high memory cost.

MQA shares K/V across all heads; GQA shares within groups.

In our experiments:

• Group-3 MQA saved significant memory.
• Slight trade-off: +2.6 PPL vs. baseline MHA.

Result Summary:

4. Hardware Architecture

The hardware pipeline consists of pre-layernorm, attention, and feed-forward network (FFN) units

4.1 Pre-LayerNorm

Computes per-channel mean/variance, approximates \(\frac{1}{\sqrt{\sigma^2 + \epsilon}}\) using a LUT-based reciprocal square root, and applies 4-bit affine parameters (γ, β).

4.2 Attention and FFN Engines

• Attention Unit:
  • Generates concatenated Q/K/V via 1×1 convolution.
  • Stores 4-bit compressed K/V in shared BRAM.
  • Computes Q–K dot-products on a PE array.
  • Applies masked softmax using exp/reciprocal LUTs.
  • Produces context vector via weighted sum over V.
• Feed-Forward Network (FFN):
  • Expands C → 4C → C.
  • Uses LUT-based GELU activation.
  • Implemented as streaming matrix-vector (mat-vec) computations using DSPs and BRAM.

4.3 Resource Utilization

5. Evaluation

• Platform: Xilinx Artix-7 100T (CSG324)
• Tool: Vivado 2018.3
• Clock Frequency: 250 MHz

The fully pipelined design sustains 1 token per cycle (after pipeline fill), reaching 4.4 K tokens/s throughput.

On-chip memory supports sequence length ≤ 256 tokens, entirely without external DRAM.

6. Conclusion and Future Work

This work demonstrates the first fully on-chip FPGA-based LLM inference using:

• 4-bit quantization,
• Multi-Query Attention, and
• A streaming Verilog architecture.

The prototype achieves real-time generation within 31.9% BRAM and 85% DSP budgets.

Future directions include:

• Scaling to larger LLMs,
• Mixed-precision inference, and
• Energy-efficiency optimization.

References

1. Vaswani et al., “Attention Is All You Need,” NeurIPS 2017.
2. Lu et al., “Hardware Accelerator for Multi-Head Attention and FFN,” IEEE SOCC 2020.
3. Keller et al., “A 95.6-TOPS/W Inference Accelerator with 4-bit Quantization in 5 nm,” IEEE JSSC 2023.
4. Zhang et al., “Algorithm-Hardware Co-Design of Attention on FPGA Devices,” ACM TECS 2021.
5. Xiao et al., “SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs,” ICML 2023.
6. Hu et al., “LoRA: Low-Rank Adaptation of LLMs,” ICLR 2022.
7. Shazeer, “Fast Transformer Decoding: One Write-Head Is All You Need,” arXiv 2019.
8. Ainslie et al., “GQA: Generalized Multi-Query Transformers,” arXiv 2023.

1. Introduction

The demand for AI processors has been rapidly increasing as machine learning applications continue to expand into edge and IoT devices. Among them, keyword spotting (KWS) and speaker verification (SV) systems have become fundamental components in modern voice-based human–machine interfaces, such as smart speakers, mobile assistants, and wearable devices. These systems enable always-on, low-power auditory perception, allowing devices to respond to wake words or authenticate users through voice features without relying on cloud processing.

In this research project, we aim to tape out a low-power AI processor, exploiting the synergy between neural model compression and hardware optimization for embedded audio intelligence.

2. Related Work

Tan et al. (JSSC 2025) [1] present a 1.8% FAR, 2 ms decision-latency SV-assisted KWS chip in 28 nm CMOS. They propose a transfer-computing model that shares convolutional layers between KWS and SV, cutting MACs and parameters by ≈1.8×. A hybrid-IF-domain circuit processes analog and digital features to eliminate ADC overhead, and a scalable 5T-SRAM reduces read/leakage power by >10×. The chip achieves 91.8 % accuracy, 1.73 nJ/decision, and 1.73 µW active power, which is the lowest among prior KWS ASICs

Xiao et al. (ISSCC 2024) [2] introduce KASP, a 55 nm KWS + SV processor using adaptive beamforming, direction-of-arrival estimation, and progressive wake-up to suppress human-voice noise. Its Lite-FDBF and Lite-X-Vector SV enable multi-user verification without speaker-specific training, reducing MACs by 99.7 % with only 1.4 % accuracy loss. KASP achieves 96.8 % KWS accuracy and 1.68 µJ/classification, representing a major advance in noise-robust, ultra-low-energy speech interfaces.

Unlike prior works, our work introduces a fully digitized computation and model architecture that enhances accuracy while leveraging early-exiting branches to reduce the switching activity of power-hungry digital cells, thereby achieving lower overall power consumption.

3. System Overview

Fig. 1 : Overall System Pipeline

The overall system pipeline, as illustrated in Fig. 1, consists of three main components: an Analog-to-Digital Converter (ADC), a Feature Extractor, and a Neural Engine.

First, the incoming acoustic waveform is captured and digitized through the ADC, which converts the continuous analog speech signal into quantized digital samples suitable for further digital processing. These quantized signals are then passed to the Feature Extractor, which computes time–frequency representations such as Mel-spectrograms to capture the spectral characteristics of the speech signal.

The extracted features are subsequently fed into the Neural Engine, which performs two parallel inference tasks: Keyword Spotting (KWS) and Speaker Verification (SV). The neural network identifies whether the spoken utterance corresponds to a predefined keyword and simultaneously verifies whether the detected voice matches a registered target speaker.

In this project, the Neural Engine (highlighted in the red box of Fig. 1) was fully implemented, including model quantization, digital inference pipeline, and hardware mapping. The ADC and Feature Extractor modules were developed by collaborating teams and interfaced with the neural engine for end-to-end system evaluation.

4. Model Design

Fig. 2 : Model Design

The proposed model architecture is based on BC-ResNet, which is known for its parameter efficiency and suitability for embedded speech applications. The original BC-ResNet consists of three main components: a CNN head, a sequence of BCBlocks, and a Classifier. To jointly handle Voice Activity Detection (VAD), Keyword Spotting (KWS), and Speaker Verification (SV) within a unified framework, the architecture was modified as shown in Fig. 2.

The input Mel-spectrogram (dimension: 1 × 1 × 39 × 59) is first processed by the CNN head, which performs initial feature extraction using convolution, batch normalization, and ReLU activation. From this output, the first three time steps of the feature map are directed to the VAD classifier, composed of four sequential convolutional blocks (Conv–BN–ReLU). If any of the three VAD logits indicate activity (i.e., vad_pred=True), the subsequent processing stages are activated.

In the next stage, the complete feature map is passed through three stacked BCBlocks, each consisting of a Conv–BN–ReLU, Conv–SSN (Sub-Spectral Normalization), Average Pooling, and an additional Conv–BN–ReLU sequence, with skip connections to preserve feature continuity and mitigate gradient degradation. These BCBlocks extract compact, discriminative embeddings suitable for both KWS and SV.

The resulting features are branched into two task-specific heads: the KWS classifier and the SV classifier. Each classifier comprises a shallow Conv–BN–ReLU stack followed by an average pooling layer, producing embedding vectors of size (1, 64) for KWS and (1, 128) for SV, respectively. These embeddings are then passed to cosine classifiers, which compute similarity scores between the embeddings and class prototypes. The index of the maximum cosine similarity determines the predicted keyword or speaker identity. For the ASIC implementation, only the neural encoder portion up to the embedding vector output was realized, while the cosine classifier was executed in software.

Quantization Setup

Quantization-aware training (QAT) was applied to reduce model precision while maintaining accuracy. Both weights and activations were quantized to 8 bits (W8A8), with 32-bit accumulations for residual connections. Each Conv–BN–ReLU and Conv–SSN pair was fused into a single quantized operation to minimize computational overhead. All layers, including AdaptiveAvgPool2d, Fully Connected, Convolution, and Residual paths, were quantized to ensure full-integer arithmetic compatibility for hardware deployment. Per-channel scaling was applied to improve numerical stability during training and inference.

Training Environment

The model was trained on the Google Speech Commands v2 (GSCD) dataset using the SGD optimizer with a momentum of 0.9 and a weight decay of 1 × 10⁻³. The total number of epochs was 300, including a 10-epoch warmup phase. The initial learning rate was set to 1.2 × 10⁻² and decayed to a lower limit of 2 × 10⁻⁵ following a cosine schedule.

Loss functions were defined using AAM-Softmax, with task-specific weighting parameters:

• SV loss (\(\lambda_{sv} = 0.3\))
• KWS loss (\(\lambda_{kws} = 0.25\))
• VAD loss (\(\lambda_{vad} = 1.0\))

Formally, the total training objective was expressed as:

where \(\mathcal{L}_{kws}\) and \(\mathcal{L}_{sv}\) were computed using \(\operatorname{AAMSoftmaxLoss}(\text{num_classes}, \text{num_emb_dim})\) for 64- and 256-dimensional embeddings, respectively.

5. Hardware Design

Fig. 3 : Hardware Architecture

The proposed hardware architecture, illustrated in Fig. 3, implements the quantized BC-ResNet model for real-time keyword spotting (KWS), speaker verification (SV), and voice activity detection (VAD). The design follows a hardware–software co-design methodology, where the quantized model described in Section 4 is mapped to a custom fixed-point inference accelerator optimized for low-power edge operation.

5.1 Architecture Overview

The entire system is implemented on-chip, consisting of a Neural Network (NN) Controller, multiple on-chip SRAM modules, a Processing Element (PE) array, and dedicated functional units. The NN Controller is hierarchically organized into submodules: Head Controller, Block Controller, VAD Controller, KWS Controller, and SV Controller, each responsible for orchestrating the corresponding layer group in the neural network.

The Feature, Weight, and Bias SRAMs store the quantized feature maps, model parameters, and biases, respectively. Each SRAM is connected to a corresponding buffer to decouple memory access from computation and enable parallel data streaming. The PE array performs core multiply–accumulate (MAC) operations, and its output passes through a Bias/Scaling Unit and a ReLU Unit before being written back to SRAM or sent to the classifier. The VAD Classifier produces the final decision, which is transmitted via an SPI interface (out_sclk, out_ss, out_sdata).

5.2 Dataflow Strategy

The accelerator employs a channel-wise (output-stationary) dataflow, where each PE is assigned to one output channel. A total of 16 PEs operate in parallel, each accumulating partial sums locally to minimize memory access overhead. The weight and feature buffers are double-buffered to enable concurrent data loading and computation, ensuring high utilization of the PE array.

This approach strikes a balance between compute efficiency and memory bandwidth, as weights remain stationary within the PEs while features are streamed from the Feature Buffer during each convolutional cycle.

5.3 Fixed-Point Arithmetic Implementation

All arithmetic operations are performed in fixed-point precision to reduce area and power consumption.

• Input to PE: 8-bit feature and 8-bit weight operands

• Output of PE: 21-bit accumulation result

  The accumulated value is then processed by the Bias/Scaling Unit, which applies per-channel scaling and bias correction to restore dynamic range. The scaled result passes through the ReLU Unit, where rounding and saturation are applied. Since ReLU6 activation is adopted, the output is saturated at a maximum value of 6, preventing numerical overflow while preserving nonlinear activation effects.

5.4 Memory Design

The on-chip memory subsystem consists of three dedicated SRAM blocks:

• Feature SRAM (64 KB) – stores intermediate activation maps
• Weight SRAM (128 KB) – holds quantized model parameters
• Bias SRAM (16 KB) – retains bias and scale coefficients

Data are transferred from the SRAMs to their respective buffers through an SPI-based write interface (wr_fsram_sclk, wr_fsram_ss, wr_fsram_sdata). The buffered memory architecture minimizes memory access latency and enables concurrent read/write operations for pipelined computation. Bandwidth analysis confirmed that the system sustains continuous dataflow at the operating frequency without stalling, even under multi-branch execution.

5.5 Clock Gating and Power Optimization

To achieve low power consumption, fine-grained clock gating was applied to idle submodules such as inactive controllers or PEs. The system dynamically disables unused PEs during lightweight tasks (e.g., VAD), reducing dynamic power dissipation. The Bias/Scaling Unit and ReLU Unit also employ operand-aware gating, activating only when valid data are available. This technique, combined with SRAM access scheduling, effectively minimizes unnecessary switching activity across the datapath.

6. ASIC Implementation

The quantized neural-network accelerator was implemented using the Samsung 28 nm Low-Power Plus (LPP) CMOS process. The full physical design flow—from RTL synthesis to place-and-route (P&R), timing closure, and sign-off—was carried out using the Synopsys Design Compiler and IC Compiler II toolchain.

6.1 Layout Overview

Fig 4. Schematic View of Synthesized Design

Fig 5. Layout view of the Design

Fig. 4 shows the post-layout view of the accelerator, including the logic and on-chip memory blocks. The core layout integrates three major SRAM macros (feature, weight, and bias memories) and the compute datapath consisting of the PE array, bias/scaling unit, and control logic. The final floorplan (Fig. 5) reveals a compact rectangular structure, with the memory macros placed symmetrically at the top and bottom of the core to minimize routing congestion and interconnect delay between compute and storage units. Metal routing was performed over six layers, and congestion analysis confirmed a utilization below 75 %, ensuring good routability.

6.2 Timing Analysis

Post-layout timing analysis was performed under typical and worst-case process–voltage–temperature (PVT) corners.

Fig. 6 : qor summary

As summarized in Fig. 6, the design achieved:

• Worst Negative Slack (WNS): 41.6 ps
• Total Negative Slack (TNS): 0 ns
• No setup or hold violations across all operating corners.

This result indicates successful timing closure with sufficient margin for 100 MHz operation under both fast–fast (1.0 V, –40 °C) and slow–slow (0.9 V, 125 °C) conditions.

6.3 Constraint and Rule Verification

Fig. 7 : Constraint verification reports

Constraint verification reports (Fig. 7) show zero violations for both maximum transition and maximum capacitance checks, confirming that all cell drive strengths and wire loads meet sign-off requirements. No DRC (Design Rule Check) or LVS (Layout Versus Schematic) violations were observed after final extraction, validating geometric and connectivity correctness.

6.4 Area Analysis

The synthesized cell area was 1.44 × 10⁶ µm², and the post-layout area increased slightly to 1.47 × 10⁶ µm² due to placement and routing overhead. The design occupied a core size of approximately 1.2 mm × 1.2 mm, which includes logic, buffers, and interconnect routing channels. The area distribution is dominated by the PE array and memory interfaces, accounting for roughly 70 % of the total area.

6.5 Power Analysis

Fig. 8 : Power BreakDown

The post-layout power breakdown obtained from PrimeTime-PX is summarized in Fig. 8.

At nominal operating conditions (0.9 V, 100 MHz):

• Total Power: 1.68 × 10⁻⁵ W (16.8 µW)
• Clock Network: 38.1 %
• Combinational Logic: 35.7 %
• Registers: 25.9 %
• Leakage Power: 55.3 % of total cell power

The relatively high leakage fraction is attributed to always-on logic and dense register networks used in the PE array. However, dynamic power remains low due to the applied clock-gating and operand-aware activation techniques described in Section 5.5.

6.6 Physical Verification and Sign-Off

The final design passed all Design Rule Check (DRC) and Layout Versus Schematic (LVS) verifications without errors. IR-drop and electromigration analyses were also performed, showing voltage droop within ±3 % of the nominal supply. The chip was thus ready for GDSII tape-out submission under the MPW program.

6.7 Summary of Implementation Results

7. Conclusion

This work presented the design and implementation of a quantized BC-ResNet–based neural network accelerator for keyword spotting (KWS) and speaker verification (SV), realized through a complete ASIC flow using the Samsung 28 nm LPP process. The proposed system adopts a hardware–software co-design approach, combining algorithmic quantization, network architecture optimization, and silicon-level efficiency enhancement to achieve real-time audio inference under strict power and area constraints.

Through quantization-aware training, the model achieved high accuracy while maintaining full 8-bit integer precision (W8A8), enabling seamless hardware mapping without floating-point overhead. The final ASIC demonstrated 16.8 µW total power consumption, a core area of 1.47 × 10⁶ µm², and timing closure at 200 kHz, validating its suitability for always-on, low-power audio processing. The architecture’s hierarchical controller structure and PE-level parallelism further ensured efficient multi-task execution across VAD, KWS, and SV domains.

The key innovations of this work include:

1. Integration of BC-ResNet architecture with quantization-aware training for efficient speech embedding generation.
2. Implementation of a fully digitized inference engine optimized for integer arithmetic and ReLU6 activation.
3. Realization of a multi-task neural accelerator capable of simultaneous voice activity detection and speaker-aware keyword recognition.

This project demonstrates the feasibility of deploying advanced neural models on resource-constrained edge platforms. Beyond this tape-out, future work will focus on post-silicon validation, multi-keyword scalability, and on-chip cosine similarity computation for complete end-to-end inference. The methodology established here provides a foundation for next-generation ultra-low-power AI processors targeting human-centric, always-on intelligence in edge and wearable devices.

8. References

[1] F. Tan, W. -H. Yu, J. Lin, K. -F. Un, R. P. Martins and P. -I. Mak, “A 1.8% FAR, 2 ms Decision Latency, 1.73 nJ/Decision Keywords-Spotting (KWS) Chip Incorporating Transfer-Computing Speaker Verification, Hybrid-IF-Domain Computing and Scalable 5T-SRAM,” in IEEE Journal of Solid-State Circuits, vol. 60, no. 3, pp. 1103-1112, March 2025

[2] J. Xiao et al., “14.8 KASP: A 96.8% 10-Keyword Accuracy and 1.68μJ/Classification Keyword Spotting and Speaker Verification Processor Using Adaptive Beamforming and Progressive Wake-Up,” 2024 IEEE International Solid-State Circuits Conference (ISSCC), San Francisco, CA, USA, 2024, pp. 268-270

Acknowledgment

This research was conducted as part of the Undergraduate Research Program (URP) and supported by the IDEC Multi-Project Wafer (MPW) initiative using Samsung 28 nm CMOS process.

https://github.com/chae-sy/capstone_design

we will present our project titled ‘Subpixel Rendering CNN Accelerator Based on Hardware–Software Co-Design Methodology.

The demand for high-resolution displays continues to grow, yet achieving high-quality visual performance remains challenging due to physical pixel density limits and fabrication constraints.

To overcome these limitations, Subpixel Rendering (SPR) technology enhances perceived resolution by manipulating not only pixels but also individual subpixels

However, conventional filter-based subpixel rendering approaches have inherent performance limitations.

To address this, we adopted a CNN-based model called SPRNN, which learns nonlinear characteristics and improves both PSNR and SSIM metrics.

Since CNNs involve heavy computation, real-time processing on mobile devices is difficult.

To solve this, we applied a hardware–software co-design strategy that combines model compression and hardware acceleration to meet both performance and power requirements.

Notably, very few previous studies have designed accelerators specialized for subpixel rendering, and even fewer have applied HW–SW co-design methodologies

Our final design target is a 45-nm ASIC that achieves:

• PSNR ≥ 30 dB, SSIM ≥ 0.8,
• Chip area ≤ 2 mm × 2 mm,
• Power consumption ≈ 800 mW, and
• Processing speed ≥ 60 FPS.”

The development schedule is shown here, and all stages were completed according to plan.

The goal of this work is to implement the Subpixel Rendering module within the Display Controller as an ASIC.

We adopted a spatial architecture that enables spatially parallel dataflow computation.

Additionally, a pipelined processing method was applied by dividing operations into multiple stages to enhance throughput.

We used Incremental Network Quantization (INQ) during CNN training to constrain weights to powers of two.

This allows multiplication operations to be replaced entirely by bit-shift operations significantly simplifying hardware.

INQ progressively quantizes the most critical weights to power-of-two values while continuing to fine-tune the remaining parameters, thus maintaining accuracy.

We implemented the CNN model identical to that in the reference paper using PyTorch.

The training achieved PSNR = 30.22 dB and SSIM = 0.86, meeting our target performance.

Next, we applied INQ-based model compression.

After retraining with power-of-two weights, the model achieved 30.01 dB PSNR, showing minimal degradation compared to the original model.

The top-level hardware diagram consists of computation modules, buffers, memory blocks, and a controller, as shown in the figure

The pipelined PE array consists of 16 superscalar PEs, each performing a 3×3 MAC operation for RGB channels over 9 cycles, producing 20-bit outputs. Both design and simulation were completed. A two-stage adder tree was designed and simulated to sum the partial results from 16 channels

The ReLU–bias module aligns a 24-bit input with a 32-bit bias via shift operations, performs bias addition, applies the ReLU activation, and truncates the result to 8 bits. The module was fully implemented and verified. We also designed and verified the Maxpool unit, which performs max-pooling operations.

The buffer temporarily stores and outputs data fetched from SRAM. The input buffer was designed to perform data-shift operations to maximize data reuse. The weight buffer supports simultaneous read and write operations, minimizing latency. The output buffer stores each channel output and performs a single write-back to SRAM once all 16 channel results are ready.

The memory stores the input feature maps, weights, and biases. The input feature memory is divided into two banks, A and B, forming a ping-pong structure that alternately stores feature maps between layers. The weight memory and bias register file store and output the respective parameters.

The top controller manages the entire system operation.

As shown in the dataflow diagram, each module is orchestrated so that input features are fetched into buffers while PEs concurrently perform 3×3 convolution over a 9-cycle pipeline, minimizing overall latency.

The left figure shows the CNN model computation mapped onto the right hardware architecture, illustrating the overall processing flow.

The top-level simulation confirmed correct memory address mapping and pipeline operation, as well as proper data transfer between layers.

Project management was conducted via GitHub, with a total of approximately 300 commits recorded.

Through quantization profiling, we analyzed the integer and fractional bit requirements for each operation.

We selected Input = int2 + frac5, Weight = int1 + frac6, and Bias = int1 + frac31 configurations. The final bit-width configuration is summarized in the table.

In the verification stage, layer-wise output comparisons confirmed that the quantization error was within ±1 LSB, corresponding to 0.027 in real value.

The synthesis result of the full SPRNN_TOP module showed a cell area of 0.189 mm² and total power consumption of 217 mW.

Comparing PE arrays implemented with multipliers versus shifters, synthesis results demonstrated an 11% area reduction and 27% power savings.

Using OpenLane, we performed automatic place-and-route (auto PnR). The layout achieved DRC zero, LVS pass, and positive setup/hold margins across all corners. (Post-layout simulation, though not included in the slides, was also confirmed as TEST PASSED.)

Finally, we will present our conclusion.

• Implemented a CNN-based Subpixel Rendering module as an ASIC.
• Maximized efficiency through multi-level parallelism and input-stationary dataflow.
• Achieved substantial area and power reductions via INQ-based shift-only computation.

The key contribution of our work lies in achieving target specifications through a hardware–software co-design approach, surpassing prior research.

Research Period: September 2, 2024 – December 20, 2024

Researcher: Seoyoon Chae

Affiliation: Sungkyunkwan University

1. Paper Review and Algorithm Design

(1) Paper Review

To investigate keyword-spotting (KWS) techniques, several papers were reviewed. Among them, one notable study proposed generating two-dimensional representations of speech signals by dividing them into frequency and time axes, then processing them through a CNN-based network [1].

The paper also described an integer-quantized approach to reduce power consumption and a custom processing-element (PE) architecture with a dedicated memory structure, optimized for low-power hardware. The study further minimized on-chip memory area by reducing SRAM size, which inspired the memory design adopted in this work.

However, the reference architecture could detect only one or two keywords, which is insufficient for large-scale KWS deployment. Therefore, in this research, the quantized bit-width was expanded from 1-bit to 8-bit, enabling the detection of five distinct keywords. The speech signal was sampled every 32 ms, and the feature-extraction circuit transformed the incoming waveform into 10 frequency bands using FFT before applying CNN operations.

The Depthwise Separable Convolutional Neural Network (DSCNN) architecture was adopted to reduce computational cost while maintaining accuracy. Although it increases the number of stages by decomposing conventional convolution into depthwise and pointwise operations, it significantly lowers overall computation and memory usage to approximately one-seventh of a standard CNN. Python-based training confirmed about 90% recognition accuracy under this structure.

(2) BC-ResNet Model Design

To extend the number of recognizable keywords to ten, several lightweight AI architectures for KWS were examined. Among them, BC-ResNet [2] demonstrated high classification accuracy with low parameter count and FLOPs, making it suitable for hardware implementation.

Using PyTorch, the smallest BC-ResNet-1 model was fine-tuned and trained on the Google Speech Commands v2 dataset, achieving 92.8% accuracy on the evaluation set.

The architecture included convolutional, batch-normalization, ReLU6, dropout, and adaptive average-pooling layers, structured as modular blocks optimized for parameter efficiency.

2. RTL Design

(1) DSCNN RTL Implementation

For hardware realization of keyword spotting, the system was designed at Register-Transfer Level (RTL) using Verilog/SystemVerilog. The architecture comprised:

• Weight Memory – stores pre-trained CNN weights
• Input Memory – receives features from the front-end
• Internal Memory – buffers intermediate results
• Controller – orchestrates operation sequencing
• PE Array – performs MAC operations (32 PEs, each 8-bit)
• Adder Tree, Output Buffer, and Comparator modules

Weights were trained offline and loaded via SPI communication. Each PE executed 8-bit integer MAC operations, and verification through testbench simulations confirmed correctness.

The Adder Tree module aggregated partial sums from PEs during pointwise or fully connected layers, implemented as a single-cycle combinational logic block to minimize latency.

The Output Buffer temporarily stored completed channel outputs and applied the ReLU activation function with saturation.

The Comparator performed a tournament-style search to identify the maximum activation value, corresponding to the detected keyword.

The Controller Module coordinated all submodules across five layers, managing memory addresses, enable/reset signals, and inter-layer synchronization.

Latency analysis showed approximately 560 clock cycles total (182 + 24 + 258 + 7 + 85 + inter-layer transitions).

(2) BC-ResNet RTL Implementation

A more advanced circuit was designed to execute BC-ResNet inference in real time. The top-level architecture included:

• PE Array (16 PEs) for convolutional MAC operations
• Normalization Module for batch normalization
• ReLU + Quantization Module for activation and 8-bit precision adjustment
• Classifier Module performing average pooling and output aggregation
• Max-Finder Module selecting the highest-scoring output label

Feature and weight data were stored in FSRAM and WSRAM, respectively, with intermediate feature/weight buffers feeding the PE array.

Control was handled by a top controller, a layer-state controller, and 23 layer controllers that managed each computational stage.

A detailed state-machine design ensured correct sequencing of weight loading, feature buffering, convolution, normalization, and activation per layer.

A frame-wise computation scheme was adopted to eliminate redundant operations between overlapping spectrogram windows (20 × 30 frequency × time). This incremental computation method [1] achieved equivalent results to full-window convolution with reduced redundancy, verified through timing analysis.

3. Logic Synthesis

After RTL verification, the design was synthesized using Synopsys Design Compiler for fabrication under Samsung 28 nm CMOS through the IDEC MPW Program.

During synthesis, non-synthesizable coding patterns (e.g., mixing blocking and non-blocking assignments, reserved identifiers, multi-signal resets) were corrected to ensure structural logic generation.

A memory wrapper cell was designed for each SRAM instance to:

1. prevent direct internal routing to memory cells, and
2. tie off unused I/O pins for electrical stability.

Post-synthesis, the top-level hierarchy was analyzed for power and timing, confirming functionality within expected constraints.

4. Place and Route (P&R)

The physical implementation was carried out using Synopsys IC Compiler II.

The flow consisted of placement, clock-tree synthesis (CTS), and routing stages.

Memory Wrapper P&R

• Imported physical libraries for memory cells provided by the foundry.
• Defined cell dimensions and placement coordinates.
• Added blockage boundaries to prevent routing through memory regions.
• Implemented power-plan connections (VDD/VSS) and guard-ring structures for stable power delivery.

Top Module P&R

• Instantiated completed memory wrappers.
• Created a mesh-style power grid across the layout.
• Performed Design Rule Check (DRC) and Layout vs Schematic (LVS) verification.

• Added I/O pad configurations and finalized GDSII files for tape-out submission.

All DRC/LVS checks were passed, and the design was submitted to the foundry for final fabrication.

References

[1] W. Shan et al., “A 510-nW Wake-Up Keyword-Spotting Chip Using Serial-FFT-Based MFCC and Binarized Depthwise Separable CNN in 28-nm CMOS,” IEEE J. Solid-State Circuits, 56(1):151–164, Jan. 2021.

[2] B. Kim et al., “Broadcasted Residual Learning for Efficient Keyword Spotting,” arXiv:2106.04140, 2023.

[3] Y. Zhang et al., “Hello Edge: Keyword Spotting on Microcontrollers,” arXiv:1711.07128, 2018.

[4] S. Choi et al., “Temporal Convolution for Real-Time Keyword Spotting on Mobile Devices,” arXiv:1904.03814, 2019.

[5] R. Tang and J. Lin, “Deep Residual Learning for Small-Footprint Keyword Spotting,” Proc. ICASSP 2018, pp. 5484–5488.

Acknowledgment

This research was conducted as part of the Undergraduate Research Program (URP) and supported by the IDEC Multi-Project Wafer (MPW) initiative using Samsung 28 nm CMOS process.

Anomaly Detection Based on 1D-CNN LSTM for Rotating Machines

Dae-Hee Lee¹, Se-Won Kim², Jung-Min Yoo³, Ye-Won Jeong⁴, Seo-Yoon Chae⁵, Chae-Gyu Lee⁶, Hyun-Seung Choo⁷, Jong-Pil Jeong⁸

¹Department of Electrical and Computer Engineering, Sungkyunkwan University

²School of Civil, Architectural and Environmental System Engineering, Sungkyunkwan University

³School of Civil, Architectural and Environmental System Engineering, Sungkyunkwan University

⁴Department of Systems Management Engineering, Sungkyunkwan University

⁵School of Electrical and Computer Engineering, Sungkyunkwan University

⁶Department of Smart Factory Convergence, Sungkyunkwan University (Co-corresponding author)

⁷Department of Electrical and Computer Engineering, Sungkyunkwan University (Co-corresponding author)

⁸Department of Smart Factory Convergence, Sungkyunkwan University (Co-corresponding author)

Abstract

The manufacturing industry is rapidly evolving in the era of the Fourth Industrial Revolution, emphasizing large-scale data, high speed, and low power consumption. Consequently, the importance of rotating machinery has significantly increased. This paper proposes a fault diagnosis method for rotating machines based on 1D-CNN LSTM (One-Dimensional Convolutional Neural Network Long Short-Term Memory). The proposed deep learning model is capable of recognizing complex patterns and making accurate predictions from large datasets. Experimental results demonstrate fault diagnosis accuracies of 95.6%, 71.4%, and 97.4% for different fault types. The proposed model enables early detection of faults, preventing production line downtime and enhancing manufacturing efficiency and competitiveness.

Keywords: Anomaly Detection, 1D-CNN LSTM, Rotating Machine, Deep Learning

I. Introduction

With the advent of the Fourth Industrial Revolution, manufacturing systems are evolving toward large-scale, high-speed, and energy-efficient operations. Rotating machinery, as a core component of industrial processes, has become increasingly critical. Any malfunction in such machinery can severely disrupt the entire production process, leading to significant economic losses. Early detection of faults through deep learning can minimize these losses and improve production efficiency, providing both industrial and economic benefits.

Recent studies have explored deep learning-based approaches such as DNNs, CNNs, and RNNs for rotating machinery fault diagnosis. Since deep learning models can learn complex patterns from large datasets, it becomes feasible to detect early-stage anomalies using data such as noise and vibration. This paper proposes a fault diagnosis method based on 1D-CNN LSTM, targeting imbalance and looseness conditions. The method aims to prevent unexpected halts in the production process, thereby improving operational efficiency and reliability.

II. Methodology

This study utilizes the rotating machinery fault-type AI dataset provided by KAMP (Korea AI Manufacturing Platform).

Figure 1 shows the Rotor Testbed, implemented using the ERA (Educational Rotor Application) Test Station from Signallink. The testbed consists of three primary modules: a Controller Module, a BLDC Motor (Brushless DC Motor), and Disks.

The Controller Module adjusts the rotational speed (RPM) of the disks; in this experiment, vibration data were collected at 1,500 RPM. The BLDC motor enables fine-grained speed control without mechanical brushes. The disk section contains 36 bolt holes, and mass imbalance was introduced by fixing a single bolt at a 270° position.

Figure 1. Rotor Testbed

The proposed 1D-CNN LSTM-based fault diagnosis system is illustrated in Figure 2.

The 1D-CNN effectively captures temporal and frequency-domain features in time-series data, while the LSTM network learns long-term dependencies inherent in such sequences. The combination of these two architectures allows efficient extraction and interpretation of temporal patterns in rotating machinery vibration data.

Figure 2. 1D-CNN LSTM Architecture

III. Experimental Results

Four categories of experimental datasets were used:

1. Normal condition (Normal)
2. Mass imbalance (Unbalance) — created by attaching a single bolt to the disk
3. Mechanical looseness (Mechanical Looseness) — simulated by partially loosening a bolt
4. Combined fault (Unbalance + Mechanical Looseness)

Each dataset contained vibration signals collected over 140 seconds. Preprocessing steps included linear interpolation, filtering, and normalization. The data were split into 240,000 training, 80,000 validation, and 80,000 test samples.

Instead of performing multi-class classification, binary classification was conducted for each fault type. Binary classification simplifies learning and reduces training time—crucial for real-time applications in manufacturing environments where rapid fault detection is essential. Moreover, since mass imbalance is the most frequent fault type in practice, it was emphasized in the modeling process.

The results are visualized in Figures 3–5 using confusion matrices. The proposed method achieved diagnostic accuracies of 95.6%, 71.4%, and 97.4% for the three fault types, respectively.

Figure 3. Normal vs. Unbalance

Figure 4. Normal vs. Mechanical Looseness

Figure 5. Normal vs. Unbalance and Mechanical Looseness

IV. Conclusion

This paper presented a 1D-CNN LSTM-based fault diagnosis system for rotating machinery using the KAMP fault-type AI dataset. The proposed architecture effectively processes time-series vibration signals by leveraging 1D-CNN for spatial pattern extraction and LSTM for learning long-term dependencies.

Experimental results demonstrated accurate classification of fault conditions, with diagnostic accuracies of 95.6%, 71.4%, and 97.4%. The model is particularly effective in detecting frequent real-world issues such as mass imbalance, making it promising for industrial deployment. The relatively lower accuracy in the unbalance dataset (71.4%) is attributed to the small amplitude variations in the vibration signals generated under this condition. Future work will extend this study to multi-class classification and further model refinement.

Acknowledgment

This research was supported by the LINC 3.0 Project, funded by the Ministry of Education and the National Research Foundation of Korea (NRF).

References

[1] Y. Huang, S. Sun, X. Duan, and Z. Chen, “A Study on Deep Neural Networks Framework,” IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), pp. 1519–1522, 2016.

[2] K. O’Shea and R. Nash, “An Introduction to Convolutional Neural Networks,” arXiv preprint arXiv:1511.08458, 2015.

[3] W. Zaremba, I. Sutskever, and O. Vinyals, “Recurrent Neural Network Regularization,” arXiv preprint arXiv:1409.2329, 2014.

[4] W. Chung, S. Lee, T. Chung, and J. Lee, “Fault Diagnosis of a Pump by Using Vibrational Signals,” Proceedings of the KSME Conference, pp. 590–595, 2001.

[5] D. Lee, H. Choo, and J. Jeong, “Anomaly Detection Based on 1D-CNN-LSTM Auto-Encoder for Bearing Data,” WSEAS Transactions on Information Science and Applications, Vol. 20, pp. 1–6, 2023.

[6] S. Kiranyaz et al., “1D Convolutional Neural Networks and Applications: A Survey,” Mechanical Systems and Signal Processing, Vol. 151, 2021.

[7] R. C. Staudemeyer and E. R. Morris, “Understanding LSTM – A Tutorial into Long Short-Term Memory Recurrent Neural Networks,” arXiv preprint arXiv:1909.09586, 2019.

[8] Korea AI Manufacturing Platform (KAMP), “Rotating Machine Fault-Type AI Dataset,” 2021. https://www.kamp-ai.kr

[9] Signallink, “Educational Rotor Application Test Station,” http://www.signallink.co.kr