Optical Deep Dive: The DSP Technology Powering AI Interconnect

Optical Deep Dive: The DSP Technology Powering AI Interconnect

In an earlier Cadence blog, Scale-Up and Scale-Out IP for Optical Interconnect for Accelerated Computing, we discussed how optical connectivity is becoming foundational to AI infrastructure. Copper remains highly effective over short distances, particularly within trays and racks, but optics becomes increasingly necessary as bandwidth, reach, and system size grow. This progression can be summarized with a simple principle:

Use Copper When You Can, Use Optical When You Must

But the transition from copper to optics is not simply a matter of replacing electrical cables with fiber. Modern optical connectivity requires a tightly integrated optoelectronic signal chain in which lasers, modulators, photodetectors, analog electronics, silicon photonics, packaging, and high-speed PHY DSP operate as one coordinated system. At 200Gbps per wavelength and beyond, DSP has become the invisible engine that turns these individual components into a reliable communications link.

From Pluggable Modules to Integrated Optics

Today’s optical interconnect options form a continuum. Full-retimed optics (FRO) use module DSPs on both transmit and receive paths for strong signal integrity and interoperability. Linear receive optics (LRO/TRO) retain transmit-side retiming, while linear pluggable optics (LPO) remove the module DSP and place greater equalization responsibility on the host PHY.

Near-packaged optics (NPO) move the optical engine close to the compute or switching ASIC, while co-packaged optics (CPO) integrate optics and host ASIC in the same package. Moving optics closer shortens high-speed electrical channels and can improve power and bandwidth density, but it also raises packaging, thermal, test, manufacturing, and serviceability challenges. The right partition depends on the complete system requirements.

Optics Moves Into the Scale-Up Domain

Optical links have supported scale-out networks for many years. The more significant transition now is the movement of optics into scale-up fabrics, where accelerators must communicate with extremely high bandwidth and low, predictable latency. As scale-up domains extend beyond a single rack, electrical reach, cable volume, bandwidth and port density, and power become increasingly difficult to address with copper alone.

To extend tightly coupled scale-up domains across multiple racks, the industry is exploring advanced optical integration approaches, including CPO and NPO. While these technologies differ in how closely optical engines are integrated with compute or switching silicon, both reflect the same architectural shift: bringing optics closer to the silicon to improve reach, bandwidth density, and power efficiency. The choice between them involves trade-offs in integration complexity, thermal management, serviceability, and ecosystem maturity.

This shift brings optics from the edge of the rack into the scale-up system itself. It does not reduce the importance of signal processing. Instead, it changes where that processing resides and expands the electrical-optical channel that the PHY must manage.

Inside the Optical Signal Chain

At first glance, an optical link appears straightforward: drive a modulator, transmit light through fiber, detect the light, and recover the data. In practice, the signal passes through several highly specialized electrical and optical components.

On the transmit side, a laser supplies continuous-wave optical power. A driver converts the outgoing electrical signal into the voltage swing required by the electro-optic modulator, which encodes the data onto the optical carrier.

In silicon-photonics implementations, this function is commonly performed by a traveling-wave Mach-Zehnder modulator (MZM) or a micro-ring resonator (MRR) modulator. The MZM manipulates the phase of light traveling through two optical paths and converts the resulting phase difference into intensity modulation. The MRR modulator electrically shifts the resonance of a compact optical ring, modulating the optical intensity as the resonance moves relative to the laser wavelength.

On the receive side, a photodetector converts incoming optical power into electrical current. Silicon-photonics platforms commonly use germanium-on-silicon photodiodes because germanium absorbs efficiently at the wavelengths used by data-center optical links and can be integrated with silicon waveguides. A transimpedance amplifier, or TIA, then converts the photodiode’s small current into a voltage suitable for equalization and data recovery.

Wavelength-division multiplexing adds another level of integration. Arrayed waveguide gratings, ring filters, couplers, splitters, modulators, and photodetectors can be incorporated into a photonic integrated circuit, or PIC. Multiple wavelengths can then share a single fiber, increasing aggregate capacity without requiring a proportional increase in fiber count.

The electronics surrounding the PIC are often referred to collectively as the electronic integrated circuit, or EIC. Depending on the implementation, the EIC may contain TIAs, modulator drivers, clocking, and control functions.

The 200G-per-Wavelength Integration Boundary

At 200G per wavelength, with signaling at approximately 100-plus GBaud using PAM4, analog bandwidth and noise requirements become demanding. TIAs and high-voltage modulator drivers have therefore often been implemented as separate devices using technologies optimized for bandwidth, gain, output swing, linearity, and noise. SiGe BiCMOS is one important implementation option, although advanced CMOS solutions are also possible.

The integration boundary, however, continues to move. Some implementations use separate DSP, driver, TIA, and photonic devices. Others integrate the modulator driver into the DSP while retaining an external TIA. More highly integrated approaches may combine both analog front ends with the DSP. Linear architectures can remove the module DSP and transfer more signal-processing responsibility to the host.

The optimal partition is not necessarily the solution with the fewest chips. It is the architecture that delivers the best end-to-end combination of signal integrity, power, latency, cost, thermal performance, manufacturability, and reliability.

Why DSP Is Central to Optical Connectivity

At 200G per wavelength, the optical channel is far from ideal. PAM4 uses four amplitude levels, creating three eyes, and the different transitions between those levels can experience unequal impairments. Receiver noise, modulator nonlinearity, photodetector response, TIA bandwidth, package loss, optical extinction characteristics, and component variation can affect those levels differently. A simple linear, level-independent channel model is therefore not always sufficient.

Optical PHY DSP addresses these challenges through functions such as transmit pre-emphasis, finite impulse response filtering, digital pre-distortion, receiver feed-forward equalization, decision-feedback equalization, clock and data recovery, level optimization, adaptive filtering, forward error correction, and link calibration. Some implementations can also use sequence-detection techniques to recover signals containing controlled residual intersymbol interference.

DSP must compensate for impairments across the complete electrical-optical-electrical signal path. This path may include the ASIC package, PCB traces, connectors, driver, modulator, fiber, photodetector, TIA, and receiver package. Electrical-channel loss, reflections, and crosstalk can close the PAM4 eyes, while optical noise, nonlinearities, bandwidth limitations, and level-dependent effects can distort them differently. Reflections can create delayed replicas of the signal and produce multipath interference, while chromatic dispersion becomes more important as baud rate and fiber reach increase, particularly when operation moves away from the zero-dispersion region.

Real systems must also operate across manufacturing variation, changing temperatures, and component aging. Laser power and wavelength can shift. Modulator bias and efficiency may drift. Photodetector responsivity, TIA gain, package characteristics, and fiber loss can vary. Adaptive PHY algorithms track these changes, update equalization and calibration parameters, and preserve link margin during operation.

The Convergence of Electrical and Optical DSP

AI infrastructure increasingly demands end-to-end signal optimization spanning both electrical and optical domains. On the electrical side, DSP must compensate for loss, reflections, and crosstalk across packages, PCB traces, connectors, backplanes, and copper cables. On the optical side, it must address noise, nonlinearities, bandwidth limitations, dispersion, and other impairments across drivers, modulators, fiber, photodetectors, TIAs, and optical packaging.

As architecture migrates from FRO to LRO, LPO, NPO, and CPO, the boundary between electrical and optical DSP becomes less distinct. In FRO, the module DSP creates a relatively clean electrical boundary between the host and the optical subsystem. In LPO, the host PHY manages a much larger end-to-end channel. In NPO and CPO, shorter electrical paths can reduce equalization requirements, but tighter integration increases the need to co-optimize SerDes, analog front ends, photonics, packaging, thermal behavior, and control.

This is why DSP is central to the optical transition. Silicon photonics provides bandwidth density. Lasers generate optical carriers. Modulators encode the data. Photodetectors convert light back into electrical signals. But DSP transforms these imperfect electrical and optical components into a manufacturable, interoperable, and reliable communications system.

As AI infrastructure expands from racks to pods and increasingly large clusters, optical connectivity will continue moving closer to compute and switching silicon. Whether a system uses FRO, LRO, LPO, NPO, or CPO, the underlying principle remains the same: the complete electrical-optical-electrical link must be optimized as one system.

The future of AI optical connectivity will therefore be shaped not only by advances in lasers, modulators, photodetectors, and silicon photonics, but also by the PHY DSP that enables these components to operate together reliably, efficiently, and at scale.