The work presented in this PhD thesis has been produced thanks to the collaboration of the teams « Digital Communications » and « Optical Telecommunications » of the department « Communications and Electronics » (COMELEC) at Telecom ParisTech in the framework of a « Futur & Ruptures » grant supported by the Fondation Telecom and the Institut Telecom.

Fiber-optic communication systems have revolutionized the telecommunications industry and have played a major role in the advent of the information age since their introduction in the 1970’s. Because of its advantages over electrical transmission, optical fibers have been largely deployed in core networks. During the last years, IPTV, HDTV, VoD, mobile broadband services and internet applications have boomed, causing saturation issues in the networks and leading to an increase in bandwidth demand. This pushed carriers to increase the capacity of WDM channels by introducing 100G and beyond systems.

With the new possibilities offered for the high speed digital circuits, coherent systems have attracted a lot of attention during the last years. Apart from the receiver sensitivity, the interest lies now in the increase of spectral efficiency as well as tolerance against dispersion effects and fiber nonlinearities. Those are today’s the most limiting factors in ultra long haul communication systems.

Additionally in contrast to Intensity Modulation Direct Detection (IM-DD) systems or differential Phase Shift Keying (PSK) systems, the received electrical signal in coherent receiver is proportional to the electrical field vector of the optical signal. Therefore the system becomes linear, which means that all linear distortions like Chromatic Dispersion (CD) and Polarization Mode Dispersion (PMD) can theoretically be compensated without any losses and also non linear effects can be compensated very efficiently. Moreover, the access to both the phase and the amplitude of the signal makes the use of advanced modulation formats such as multilevel formats such as Quadrature PSK (QPSK) and Quadrature Amplitude Modulation (QAM), which can raise the spectral efficiency up to several bit/s/Hz, compared to only 1bit/s/Hz for IM-DD systems. However, these advantages have their price. A coherent receiver is much more complex than a simple direct detection receiver. High speed analog to digital converters (ADC) are needed to convert the received signal into the digital domain. And these ADCs must be interfaced with a digital signal processing unit, which performs polarization control, equalization and finally the carrier and data recovery. To develop these components and algorithms for the next generation of optical communication systems, which can run at 100G and beyond is still a challenging issue.

The aim of this thesis is to develop robust Digital Signal Processing (DSP) tools specific to the optical channel. Those algorithms at least deal with the operations of carrier phase recovery, frequency offset estimation, equalization and the tracking of the variation of PMD. Knowing that those algorithms are to be implemented in circuits running with a speed of some tens of MHz, the optimization of those tools is essential. Moreover, the already proposed tools are well adapted for QPSK formats, and as higher order modulation formats are more sensitive to signal distortions, accurate estimators and robust equalizers are still required for such QAM formats. Our proposed algorithms should be tested using a simulation setup of an optical transmission system using coherent detection and validated with offline processing of real measurements in the framework of the European Network of Excellence (EURO-FOS).

State-of-the-art of optical transmission systems

Optical transmission systems have been evolving in the last three decades in terms of capacity, reach and the structure of networks. In the 1980’s, single span fibers were used to transmit of ∼ 622Mbit/s. In the 1990’s, the development of Erbium Doped Fiber Amplifier (EDFA) allowed the use of multi span fibers and the increase of the transmission distance. Moreover, the Wavelength Division Multiplexing (WDM) was introduced. In fact, multiplexing different channels in a single fiber where each channel is centered on a specific wavelength, ensures high transmission capacity and lowers the cost per transmitted bit. Such systems were operating at a rate of 2.5Gbit/s per WDM channel.

In the year 2000’s, along with the continuous deployment of dense WDM (DWDM) systems, mesh based architectures for traffic were introduced in order to achieve better capacity efficiency. This was made possible through the use of intelligent network elements such as Optical cross-connects or the Reconfigurable Optical Add and Drop Multiplexers (ROADM). Typically, those systems operate at a rate of 10Gbit/s per channel and are widely used nowadays. Telecom Fibers are used for a wide range of applications including the access (distances < 100km), Metropolitan (100−300km), Regional (300−1000km) and (Ultra) long-haul (>1000km) terrestrial or submarine transmission. In the framework of this thesis, we focus on terrestrial long haul transmission using optical amplification. For this kind of systems, the C-band ranging from 1528.77nm and 1568.36nm is generally used. This C-band corresponds to the minimum attenuation of the fiber .

the grid has been extended to 50GHz and 25GHz spaced grids. Today, most deployed optically-routed mesh networks use 80 WDM channels centered on the 50GHz ITU grid, each channel is of capacity 10Gbit/s using an (Non-return-to-Zero) NRZ On-Off Keying (OOK) modulation format. The total capacity of the fiber link is therefore 0.8Tbit/s across 2000km.

The second option is actually a promising method to increase the capacity of the existing systems. Indeed, during the last 15 years, this capacity has increased more than three times, proportionally to the bandwidth demand [7]. This growth has been  carried out through increasing the baud rate of the intensity modulated direct detection (IM-DD) signal and the number of the DWDM channels inside the C-band. Another solution may consist in using multilevel intensity modulated systems. We discuss now these three alternatives:

• Reducing the channel spacing inside C-band: Adding more channels raises two main problems: coherent WDM crosstalk and filter narrowing [1, 8]. The coherent WDM crosstalk is due to interference of the received optical signal field and the residual optical field of a neighboring channel. This leads to signal distortion after WDM demultiplexing and system penalties. The filter narrowing is the fact that the concatenation of several multiplexingdemultiplexing filters in today’s mesh routed optical networks narrows the overall (equivalent) filter and distorts the signal.

• Using Multilevel intensity modulation: M-ary Amplitude Shift Keying have showed poor performance for fiber-optic transmission so far, mainly due to important back-to-back receiver sensitivity penalty compared to binary OOK [1]. For example, 4-ASK produce a penalty of about 8dB.

• Increasing the Baud rate of IM-DD: The intensity modulation system is widely used because of its relative tolerance to the Amplified Spontaneous Emission (ASE) noise and the simplicity of the modulators and the receivers. The receivers use the Direct Detection scheme with a threshold detector.