Acoustic invariance in speech production: evidence from measurements of the spectral characteristics of stop consonants. On the basis of theoretical considerations and the results of experiments with synthetic consonant-vowel syllables, it has been hypothesized that the short-time spectrum sampled at the onset of a stop consonant should exhibit gross properties that uniquely specify the consonantal place of articulation independent of the following vowel. The aim of this paper is to test this hypothesis by measuring the spectrum sampled at the onsets and offsets of a large number of consonant-vowel (CV) and vowel-consonant (VC) syllables containing both voiced and voiceless stops produced by several speakers. Templates were devised in an attempt to capture three classes of spectral shapes: diffuse-rising, diffuse-falling, and compact, corresponding to alveolar, labial, and velar consonants, respectively. Spectra were derived from the utterances by sampling at the consonantal release of CV syllables and at the implosion and burst release of VC syllables, and these spectra (smoothed by a linear prediction algorithm) were matched against the templates.

Evaluation of two voice-separation algorithms using normal-hearing and hearing-impaired listeners. Two signal-processing algorithms, designed to separate the voiced speech of two talkers speaking simultaneously at similar intensities in a single channel, were compared and evaluated. Both algorithms exploit the harmonic structure of voiced speech and require a difference in fundamental frequency (F0) between the voices to operate successfully. One attenuates the interfering voice by filtering the cepstrum of the combined signal. The other uses the method of harmonic selection [T. W. Parsons, J. Acoust. Soc. Am. 60, 911-918 (1976)] to resynthesize the target voice from fragmentary spectral information. Two perceptual evaluations were carried out. One involved the separation of pairs of vowels synthesized on static F0's; the other involved the recovery of consonant-vowel (CV) words masked by a synthesized vowel. Normal-hearing listeners and four listeners with moderate-to-severe, bilateral, symmetrical, sensorineural hearing impairments were tested. All listeners showed increased accuracy of identification when the target voice was enhanced by processing.

Vowel-onset detection. An algorithm is presented that correctly detects the large majority of vowel onsets in fluent speech. The algorithm is based on the simple assumption that vowel onsets are characterized by the appearance of rapidly increasing resonance peaks in the amplitude spectrum. Application to carefully articulated, isolated words results in a high number of false alarms, predominantly before consonants that can function as vowels in a different context such as another language or as a syllabic consonant. After applying some modifications in the setting of some parameters, this number of false alarms for isolated words can be reduced significantly, without the risk of a large number of missed detections. The temporal accuracy of the algorithm is better than 20 ms. This accuracy is determined with respect to the perceptual moment of occurrence of a vowel onset as determined by a phonetician. Published in The Journal of the Acoustical Society of America (1990)

Structural design of hidden Markov model speech recognizer using multivalued phonetic features: comparison with segmental speech units. A novel approach to speech recognition, on the basis of a multidimensional multivalued phonetic-feature description of speech signals, is presented and evaluated. The hidden Markov model (HMM) framework is used to provide the recognition algorithm, which assumes that the underlying Markov chain tracks the temporal evolution of the features. It is shown that this approach can naturally accommodate such coarticulatory effects as feature spreading and formant transition in the functionality of the recognizer, and can provide a high degree of acoustic data sharing that makes effective use of training data. Use of phonetic features as the basic speech units creates a framework where the Markov model's state topology in the recognizer can be designed with guidance of detailed speech knowledge. Details of such a design for a stop consonant-vowel vocabulary are described.

Naturalistic recordings capture audio in real-world environments where participants behave naturally without interference from researchers or experimental protocols. Naturalistic long-form recordings extend this concept by capturing spontaneous and continuous interactions over extended periods, often spanning hours or even days, in participants' daily lives. Naturalistic recordings have been extensively used to study children's behaviors, including how they interact with others in their environment, in the fields of psychology, education, cognitive science, and clinical research. These recordings provide an unobtrusive way to observe children in real-world settings beyond controlled and constrained experimental environments. Advancements in speech technology and machine learning have provided an initial step for researchers to automatically and systematically analyze large-scale naturalistic recordings of children. Despite the imperfect accuracy of machine learning models, these tools still offer valuable opportunities to uncover important insights into children's cognitive and social development. Several critical speech technologies involved include speaker diarization, vocalization classification, word count estimate from adults, speaker verification, and language diarization for code-switching. Most of these technologies have been primarily developed for adults, and speech technologies applied to children specifically are still vastly under-explored. To fill this gap, we discuss current progress, challenges, and opportunities in advancing these technologies to analyze naturalistic recordings of children during early development (< 3 years of age). We strive to inspire the signal processing community and foster interdisciplinary collaborations to further develop this emerging technology and address its unique challenges and opportunities.

formants, adjustment of vibrato, and adjustments to vowels and consonants. Sample libraries for various languages and various accents are available. With Digital music technology encompasses the use of digital instruments to produce, perform or record music. These instruments vary, including computers, electronic effects units, software, and digital audio equipment. Digital music technology is used in performance, playback, recording, composition, mixing, analysis and editing of music, by professions in all parts of the music industry. At Bell Laboratories, Max Matthews worked with researchers Kelly and Lochbaum to develop a model of the vocal tract to study how its properties contributed to speech generation. Using the model of the vocal tract,—a method, which would come to be known as physical modeling synthesis, in which a computer estimates the formants and spectral content of each word based on information about the vocal model, including various… In the 2010s, singing synthesis technology took advantage of the advances in artificial intelligence, deep listening and machine learning, to better represent the nuances of the human voice. New high-fidelity sample libraries combined with digital audio workstations facilitate editing in fine detail, such as shifting of formants, adjustment of vibrato, and adjustments to vowels and consonants. Sample libraries for various languages and various accents are available. With advancements in vocal synthesis, artists sometimes use sample libraries in lieu of backing singers. A synthesizer is an electronic musical instrument that generates electric signals that are converted to sound through instrument amplifiers and loudspeakers or headphones. Synthesizers may either imitate existing sounds (instruments, vocal, natural sounds, etc.), or generate new electronic timbres or sounds that did not exist before. They are often played with an electronic musical keyboard, but they can be controlled via a variety of other input de At Bell Laboratories, Max Matthews worked with researchers Kelly and Lochbaum to develop a model of the vocal tract to study how its properties contributed to speech generation. Using the model of the vocal tract,—a method, which would come to be known as physical modeling synthesis, in which a computer estimates the formants and spectral content of each word based on information about the vocal model, including various applied filters representing the vocal tract—to make a computer (an IBM 704) sing for the first time in 1962. The computer performed a rendition of "Daisy Bell". In the 2010s, singing synthesis technology took advantage of the advances in artificial intelligence, deep listening and machine learning, to better represent the nuances of the human voice. New high-fidelity sample libraries combined with digital audio workstations facilitate editing in fine detail, such as shifting of formants, adjustment of vibrato, and adjustments to vowels and consonants. Sample libraries for various languages and various accents are available. With advancements in vocal synthesis, artists sometimes use sample libraries in lieu of backing singers. A synthesizer is an electronic musical instrument that generates electric signals that are converted to sound through instrument amplifiers and loudspeakers or headphones. Synthesizers may either imitate existing sounds (instruments, vocal, natural sounds, etc.), or generate new electronic timbres or sounds that did not exist before. They are often played with an electronic musical keyboard, but they can be controlled via a variety of other input devices, including sequencers, instrument controllers, fingerboards, guitar synthesizers, wind controllers, and electronic drums. Synthesizers without built-in controllers are often called sound modules, and are controlled using a controller device. Synthesizers use various methods to generate a signal. Among the most popular waveform synthesis techniques are subtractive synthesis, additive synthesis, wavetable synthesis, frequency modulation synthesis, phase distortion synthesis, physical modeling synthesis and sample-based synthesis or a variant, granular synthesis. Synthesizers are used in many genres of pop, rock and dance music. Contemporary classical music composers from the 20th and 21st centuries write compositions for synthesizer. Sampling has its roots in France with the sound experiments carried out by musique concrète practitioners. Digital sampling technology, introduced in the 1970s, has become a staple of music production in the 2000s. Devices that use sampling, record a sound digitally (often a musical instrument, such as a piano or flute being played), and replay it when a key or pad on a controller device (e.g., an electronic keyboard, electronic drum pad, etc.) is pressed or triggered. Samplers can alter the sound using various audio effects and audio processing. In the 1980s, when the technology was still in its infancy, digital samplers cost tens of thousands of dollars and they were only used by the top recording studios and musicians. These were out of the price range of most musicians. Early samplers include the 8-bit Electronic Music Studios MUSYS-3 circa 1970, Computer Music Melodian in 1976, Fairlight CMI in 1979, Emulator I in 1981, Synclavier II Sample-to-Memory (STM) option circa 1980, Ensoniq Mirage in 1984, and Akai S612 in 1985. The latter's successor, the Emulator II (released in 1984), listed for US$8,000 equivalent to $24,792 in 2025. Other samplers were released during this period with high price tags, such as the K2000 and K2500. Some important hardware samplers include the Kurzweil K250, Akai MPC60, Ensoniq Mirage, Ensoniq ASR-10, Akai S1000, E-mu Emulator, and Fairlight CMI. One of the biggest uses of sampling technology was by hip-hop music DJs and performers in the 1980s. Before affordable sampling technology was readily available, DJs would use a technique pioneered by Grandmaster Flash to

Although Singing Voice Synthesis (SVS) has revolutionized audio content creation, global linguistic diversity remains challenging. Current SVS research shows scant exploration of cross-lingual generalization, as fragmented, language-specific phoneme encodings (e.g., Pinyin, ARPA) hinder unified phonetic modeling. To address this challenge, we built a four-language dataset based on GTSinger's speech data, using the International Phonetic Alphabet (IPA) for consistent phonetic representation and applying precise segmentation and calibration for improved quality. In particular, we propose a novel method of decomposing IPA phonemes into letters and diacritics, enabling the model to deeply learn the underlying rules of pronunciation and achieve better generalization. A dynamic IPA adaptation strategy further enables the application of learned phonetic representations to unseen languages. Based on VISinger2, we introduce Transinger, an innovative cross-lingual synthesis framework. Transinger achieves breakthroughs in phoneme representation learning by precisely modeling pronunciation, which effectively enables compositional generalization to unseen languages. It also integrates Conformer and RVQ techniques to optimize information extraction and generation, achieving outstanding cross-lingual synthesis performance. Objective and subjective experiments have confirmed that Transinger significantly outperforms state-of-the-art singing synthesis methods in terms of cross-lingual generalization. These results demonstrate that multilingual aligned representations can markedly enhance model learning efficacy and robustness, even for languages not seen during training. Moreover, the integration of a strategy that splits IPA phonemes into letters and diacritics allows the model to learn pronunciation more effectively, resulting in a qualitative improvement in generalization. Moreover, the integration of a strategy that splits IPA phonemes into letters and diacritics allows the model to learn pronunciation more effectively, resulting in a qualitative improvement in generalization. Keywords: voice synthesis, singing voice synthesis, audio signal analysis, artificial intelligence, phonetics, cross-lingual, audio processing, deep generative models status released display-pdf yes is-olf no is-manuscript no is-preprint no is-journal-matter no is-scanned no is-retracted no Received 2025 Apr 15; Revised 2025 Jun 14; Accepted 2025 Jun 24; Collection date 2025 Jul. 1. Compared to traditional single-stage quantization methods, RVQ’s progressive residual learning mechanism preserves language-shared underlying articulation patterns (e.g., consonant articulation positions, vowel formant distributions) while disentangling language-specific acoustic details (e.g., pitch modulation in tonal languages). This hierarchical codebook structure significantly enhances parameter efficiency, enabling the model to generalize to linguistically related new language variants using only limited training data. Audio Codec The evolution of audio coding has progressed from traditional hybrid architectures to modern neural end-to-end paradigms, significantly impacting singing voice compression capabilities. Conventional hybrid codecs like Opus [ 19 ] and Adaptive Multi-Rate (AMR) [ 20 ] combine waveform-preserving and parametric techniques through manual engineering. Opus’s dual-codec approach (SILK for speech/CELT for music) achieves balanced quality–latency tradeoffs, while AMR’s Algebraic Code-Excited Linear Prediction (ACELP) enables adaptive speech encoding. Recent breakthroughs in deep learning have accelerated neural codec development through three key innovations: (1) training scale expansion from constrained speech corpora to diverse audio collections encompassing singing voices, enabling robust cross-domain generalization; (2) discrete tokenization via the Vector-Quantized Variational Autoencoder (VQ-VAE) [ 21 ], which facilitates structured latent representations crucial for encoding singing voice harmonics; and (3) integration of language model-inspired architectures using self-attention mechanisms that effectively capture long-range pitch dependencies and formant relationships in vocal signals. English, in contrast, is stress-timed and characterized by frequent consonant onsets and syllabic stress alternations, requiring the model to capture dynamic prosodic variation and complex phonotactics. These differences result in diverse acoustic manifestations across languages, significantly complicating efforts to build a unified SVS model. Our dataset also reflects substantial phonetic variation at the segmental level: retroflexes and tone markers dominate Chinese samples, nasalized vowels are Prior Encoder To effectively capture phonetic characteristics, the text encoding module in the prior encoder embeds IPA letters and diacritics separately and combines them through element-wise addition. This method ensures flexible phoneme encoding while preserving distinct phonetic features, enabling the model to learn separate feature representations for each component. Consequently, the model enhances phoneme encoding accuracy and flexibility and is better equipped to recognize and generate phonemes not encountered during training. Furthermore, this decomposition strategy significantly improves modeling efficiency and generalization. In the context of phonetic alignment, this encourages consistency in semantic identity—particularly important for capturing phoneme categories (e.g., vowels vs. consonants) and for supporting IPA-based decomposition strategies, where both symbol and modifier components must align directionally for accurate sub-phoneme modeling. (3) L cos = 1 − ∑ i = 1 N q prior , i · q posterior , i ∥ q prior , i ∥ ∥ q posterior , i ∥ L1 loss computes the absolute element-wise differences between the prior and posterior quantizations. Compared to MSE, L1 loss is less sensitive to outliers and extreme deviations.