The Speech Signal Improvement Challenge Grand Challenge at ICASSP 2024 is intended to stimulate research in the area of improving the speech signal quality in communication systems. The speech signal quality is measured with SIG in ITU-T P.835 and is still a top issue in audio communication and conferencing systems.<\/p>\n\n\n\n
This challenge is to benchmark the performance of real-time speech enhancement models with a real (not simulated) test set. The audio scenario is the send signal in telecommunication; it does not include echo impairments. Participants will evaluate their speech enhancement model on a test set and submit the results (clips) for evaluation.<\/p>\n\n\n\n
Challenge tracks<\/h3>\n\n\n\n
There are two tracks for this challenge:<\/p>\n\n\n\n
- \n
- Real-time track<\/li>\n\n\n\n
- Non-real-time track<\/li>\n<\/ol>\n\n\n\n
Latency and runtime requirements<\/h3>\n\n\n\n
Algorithmic <\/em><\/strong>laten<\/em><\/strong>cy:<\/em><\/strong> The offset introduced by the whole processing chain including STFT, iSTFT, overlap-add, additional lookahead frames, etc., compared to just passing the signal through without modification. But this doesn\u2019t include buffering latency.<\/p>\n\n\n\n
- \n
- Ex.1: A STFT-based processing with window length = 20 ms and hop length = 10 ms introduces an algorithmic delay of window length \u2013 hop length = 10 ms.<\/li>\n\n\n\n
- Ex.2: A STFT-based processing with window length = 32 ms and hop length = 8 ms introduces an algorithmic delay of window length \u2013 hop length = 24 ms.<\/li>\n\n\n\n
- Ex.3: An overlap-save-based processing algorithm introduces no additional algorithmic latency.<\/li>\n\n\n\n
- Ex.4: A time-domain convolution with a filter kernel size = 16 samples introduces an algorithmic latency of kernel size \u2013 1 = 15 samples. Using one-sided padding, the operation can be made fully \u201ccausal\u201d, i.e. a left-sided padding with kernel size-1 samples would result in no algorithmic latency.<\/li>\n\n\n\n
- Ex.5: A STFT-based processing with window_length = 20 ms and hop_length = 10 ms using 2 future frames information introduce an algorithmic latency of (window_length \u2013 hop_length) + 2*hop_length = 30 ms.<\/li>\n<\/ul>\n\n\n\n
Buffering latency:<\/em><\/strong> It is defined as the latency introduced by block-wise processing, often referred to as hop-size, frame-shift, or temporal stride.<\/p>\n\n\n\n
\u2022 Ex.1: A STFT-based processing has a buffering latency corresponding to the hop size
\u2022 Ex.2: A overlap-save processing has a buffering latency corresponding to the frame size.
\u2022 Ex.3: A time-domain convolution with stride 1 introduces a buffering latency of 1 sample.<\/p>\n\n\n\nReal-time factor (RTF):<\/em><\/strong> RTF is defined as the fraction of time it takes to execute one processing step. For a STFT-based algorithm, one processing step is the hop-size. For a time-domain convolution, one processing step is 1 sample. RTF = compute time\/time step.<\/p>\n\n\n\n
All models submitted to this challenge must meet all of the below requirements.<\/strong><\/p>\n\n\n\n
- \n
- For the real-time track: To be able to execute an algorithm in real-time, and to accommodate for variance in compute time which occurs in practice, we require RTF <= 0.5 in the challenge on an Intel Core i5 Quadcore clocked at 2.4 GHz using a single thread. The non-real-time track must have a RTF > 0.5.<\/li>\n\n\n\n
- Algorithmic latency + buffering latency <= 20ms.<\/li>\n\n\n\n
- No future information can be used during model inference.<\/li>\n<\/ol>\n\n\n\n
Registration procedure<\/h3>\n\n\n\n
To register for the challenge,\u202fparticipants are required to email Speech Signal Improvement Challenge sig_challenge@microsoft.com (opens in new tab)<\/span><\/a> with the name of their team members, emails, affiliations, team name, track(s) participating in, team captain, and tentative paper title. Participants also need to register on the Challenge CMT (opens in new tab)<\/span><\/a> site where they can submit the enhanced clips. Registration data is captured and stored in the US.<\/p>\n\n\n\n
Submission instructions<\/h3>\n\n\n\n
Please use Microsoft Conference Management Toolkit (opens in new tab)<\/span><\/a> for submitting the results. After logging in, complete the following steps to submit the results:<\/p>\n\n\n\n
- \n
- Choose \u201cCreate new submission\u201d in the Author Console.<\/li>\n\n\n\n
- Enter the title, abstract, and co-authors, and upload a lastname<\/em>.txt file (can be empty or contain additional information regarding the submission).<\/li>\n\n\n\n
- Compress the enhanced results files to a single lastname<\/em>.zip file, retaining the same folder and file names as the blind test set (max file size: 1.8 GB).<\/li>\n\n\n\n
- After creating the submission, return to the \u201cAuthor Console\u201d (by clicking on \u201cSubmissions\u201d at the top of the page) and upload the lastname<\/em>.zip file via \u201cUpload Supplementary Material\u201d.<\/li>\n<\/ol>\n\n\n\n
Contact us:<\/strong> For questions, please contact sig_challenge@microsoft.com<\/a><\/p>\n\n\n\n
<\/p>\n\n\n\n\n\n
Official rules<\/h2>\n\n\n\n
Sponsor<\/h3>\n\n\n\n
These Official Rules (\u201cRules\u201d) govern the operation of the Microsoft ICASSP 2024 Signal Enhancement (see overview) Event Contest (\u201cContest\u201d). Microsoft Corporation, One Microsoft Way, Redmond, WA, 98052, USA, is the Contest sponsor (\u201cSponsor\u201d).<\/p>\n\n\n\n
Definitions<\/h3>\n\n\n\n
In these Rules, \u201cMicrosoft\u201d, \u201cwe\u201d, \u201cour\u201d, and \u201cus\u201d, refer to Sponsor and \u201cyou\u201d and \u201cyourself\u201d refers to a Contest participant, or the parent\/legal guardian of any Contest entrant who has not reached the age of majority to contractually obligate themselves in their legal place of residence. \u201cEvent\u201d refers to the ICASSP 2024 Speech Signal Improvement Challenge (the \u201cEvent\u201d). By entering you (your parent\/legal guardian if you are not the age of majority in your legal place of residence) agree to be bound by these Rules.<\/p>\n\n\n\n
Entry period<\/h3>\n\n\n\n
The Contest will operate from October 15, 2023 to January 23, 2024 (\u201cEntry Period\u201d). The Entry Period is divided into several periods as described in section How to Enter.<\/p>\n\n\n\n
Eligibility<\/h3>\n\n\n\n
Open to any registered Event attendee 18 years of age or older. If you are 18 years of age or older but have not reached the age of majority in your legal place of residence, then you must have consent of a parent\/legal guardian. Employees and directors of Microsoft Corporation and its subsidiaries, affiliates, advertising agencies, and Contest Parties are not eligible, nor are persons involved in the execution or administration of this promotion, or the family members of each above (parents, children, siblings, spouse\/domestic partners, or individuals residing in the same household). Void in Cuba, Iran, North Korea, Sudan, Syria, Region of Crimea, and where prohibited. For business\/tradeshow events: If you are attending the Event in your capacity as an employee, it is your sole responsibility to comply with your employer\u2019s gift policies. Microsoft will not be party to any disputes or actions related to this matter.<\/p>\n\n\n\n
How to enter<\/h3>\n\n\n\n
The Contest Objective is to promote collaborative research in real-time single-channel Speech Enhancement aimed to maximize the subjective (perceptual) quality of the enhanced speech. Winners will be determined based on the speech quality of signal enhancement models using the online subjective evaluation framework ITU-T P.804. Only models described in accepted ICASSP 2024 Grand Challenge papers will be eligible for winning the Contest. See signal_paper<\/em> for additional Contest details. You may participate as an individual or a team. If forming a team, you must designate a \u201cTeam Captain\u201d who will submit all entry materials on behalf of the team. Once you register as part of a Team, you cannot change Teams or alter your current team (either by adding or removing members) after the submission of your Entry. Limit one Entry per person and per team. You may not compete on multiple teams and you may not enter individually and on a team. We are not responsible for Entries that we do not receive for any reason, or for Entries that we receive but are not decipherable or not functional for any reason. Each Team is solely responsible for its own cooperation and teamwork. In no event will Microsoft officiate in any dispute regarding the conduct or cooperation of any Team or its members. The Contest will operate as follows: Registration \/ Development Period: October 15, 2023 to December 5, 2023. To register, please send an email to sig_challenge@microsoft.com<\/a> stating that you are interested to participate in the challenge. Please include the following details in your email:<\/p>\n\n\n\n
- \n
- Names of the participants and name of the team captain<\/li>\n\n\n\n
- Institution\/Company<\/li>\n\n\n\n
- Email<\/li>\n<\/ul>\n\n\n\n
Then, i. develop a speech enhancement model that best meets the Contest Objective as described more fully at signal_paper<\/em> and ii. submit a ICASSP 2024 Grand Challenge paper via Microsoft Conference Management Toolkit(opens in new tab) (opens in new tab)<\/span><\/a> which reports the computational complexity of the model in terms of the number of parameters and the time it takes to infer a frame on a particular CPU (preferably Intel Core i5 quad core machine clocked at 2.4 GHz). To develop your model, use any publicly available dataset for training data, including the Contest datasets provided for training and developing models. You may augment your datasets with the Contest dataset. You can augment your data in any way that improves the performance of your model. The final evaluation will be conducted on a blind test set that is similar to the open sourced test set. Testing \/ Entry Period: December 10-17, 2023. On December 5, 2023 the blind test dataset will be made available. You will have until 11:59 PM PT on December 10, 2023 to test your model against this dataset and create a set of enhanced clips to submit for judging (your \u201cEntry\u201d). The rules of the challenge are as follows:<\/p>\n\n\n\n
- \n
- For real-time track, the processing must take less than the stride time T_s<\/em> (in ms) to process a frame of size T<\/em> (in ms) on an Intel Core i5 quad-core machine clocked at 2.4 GHz or equivalent processors. For example, T_s = T\/2<\/em> for 50% overlap between frames.<\/li>\n\n\n\n
- For non-real-time track, there are no constraints on computation time.<\/li>\n\n\n\n
- Algorithmic latency + buffering latency <= 20ms<\/li>\n\n\n\n
- Signal enhancement can be done via a deep model, a traditional signal processing algorithm, or a mix of the two. There are no restrictions on the processing aside from the run time and algorithmic delay described above.<\/li>\n\n\n\n
- Submissions must follow instructions on overview.<\/li>\n\n\n\n
- Winners will be picked based on the subjective MOS evaluated on the blind test set using P.804 (listening phase) \/ P.863.2 framework.<\/li>\n\n\n\n
- The blind test set will be made available to the participants on December 10, 2023. Participants must send the results (audio clips) achieved by their developed models to the organizers. We will use the submitted clips to conduct subjective evaluation and pick the winners based on the results. Participants are forbidden from using the blind test set to retrain or tune their models. They should not submit results using other processing methods that they are not submitting to ICASSP 2024. Failing to adhere to these rules will lead to disqualification from the challenge.<\/li>\n\n\n\n
- Participants should report the computational complexity of their model in terms of the number of parameters and the time it takes to infer a frame on a particular CPU (preferably Intel Core i5 quad-core machine clocked at 2.4 GHz).<\/li>\n\n\n\n
- Each participating team must submit an ICASSP paper that summarizes the research efforts and provide all the details to ensure reproducibility. Authors may choose to report additional objective\/subjective metrics in their paper. The papers need to be in ICASSP format as outlined in\u00a0ICASSP 2024 Paper Kit (opens in new tab)<\/span><\/a> (opens in new tab)<\/span><\/a><\/li>\n\n\n\n
- Submitted papers will undergo the standard peer-review process of ICASSP 2024. The paper needs to be accepted to the conference for the participants to be eligible for the challenge.<\/li>\n\n\n\n
- You agree to not attempt to determine the identity of speakers in the Common Voice dataset.<\/li>\n<\/ul>\n\n\n\n
ICASSP 2024 Grand Challenge Paper Submission and Judging Period: December 17, 2023 \u2013 11:59 PM PT January 2, 2024. To submit a paper, visit Microsoft Conference Management Toolkit(opens in new tab) (opens in new tab)<\/span><\/a>. The entry limit is one per person during the Entry Period. Any attempt to obtain more than the stated number of entries by using multiple\/different accounts, identities, registrations, logins, or any other methods will void your entries and you may be disqualified. Use of any automated system to participate is prohibited. We are not responsible for excess, lost, late, or incomplete entries. If disputed, entries will be deemed submitted by the \u201cauthorized account holder\u201d of the email address, social media account, or other method used to enter. The \u201cauthorized account holder\u201d is the natural person assigned to an email address by an internet or online service provider, or other organization responsible for assigning email addresses.<\/p>\n\n\n\n
Eligible entry<\/h3>\n\n\n\n
To be eligible, an entry must meet the following content\/technical requirements:<\/p>\n\n\n\n
- \n
- Your Entry must be the method described in a paper accepted as ICASSP 2024 Grand Challenge paper.<\/li>\n\n\n\n
- Your entry must be your own original work; and<\/li>\n\n\n\n
- Your entry cannot have been selected as a winner in any other contest; and<\/li>\n\n\n\n
- You must have obtained any and all consents, approvals, or licenses required for you to submit your entry; and<\/li>\n\n\n\n
- To the extent that entry requires the submission of user-generated content such as software, photos, videos, music, artwork, essays, etc., entrants warrant that their entry is their original work, has not been copied from others without permission or apparent rights, and does not violate the privacy, intellectual property rights, or other rights of any other person or entity. You may include Microsoft trademarks, logos, and designs, for which Microsoft grants you a limited license to use for the sole purposes of submitting an entry into this Contest; and<\/li>\n\n\n\n
- Your entry may NOT contain, as determined by us in our sole and absolute discretion, any content that is obscene or offensive, violent, defamatory, disparaging or illegal, or that promotes alcohol, illegal drugs, tobacco or a particular political agenda, or that communicates messages that may reflect negatively on the goodwill of Microsoft.<\/li>\n\n\n\n
- Your entry must NOT include enhanced clips using other signal enhancement methods that you are not submitting to ICASSP 2024.<\/li>\n<\/ul>\n\n\n\n
Use of entries<\/h3>\n\n\n\n
We are not claiming ownership rights to your Submission. However, by submitting an entry, you grant us an irrevocable, royalty-free, worldwide right and license to use, review, assess, test and otherwise analyze your entry and all its content in connection with this Contest and use your entry in any media whatsoever now known or later invented for any non-commercial or commercial purpose, including, but not limited to, the marketing, sale or promotion of Microsoft products or services, without further permission from you. You will not receive any compensation or credit for use of your entry, other than what is described in these Official Rules. By entering you acknowledge that the we may have developed or commissioned materials similar or identical to your entry and you waive any claims resulting from any similarities to your entry. Further you understand that we will not restrict work assignments of representatives who have had access to your entry and you agree that use of information in our representatives\u2019 unaided memories in the development or deployment of our products or services does not create liability for us under this agreement or copyright or trade secret law. Your entry may be posted on a public website. We are not responsible for any unauthorized use of your entry by visitors to this website. We are not obligated to use your entry for any purpose, even if it has been selected as a winning entry.<\/p>\n\n\n\n
Winner selection and notification<\/h3>\n\n\n\n
Pending confirmation of eligibility, potential winners will be selected by Microsoft or their Agent or a qualified judging panel from among all eligible entries received based on the following judging criteria: 99% \u2013 The subjective speech quality evaluated on the blind test set using ITU-T P.804 framework. We will use the submitted clips with no alteration to conduct ITU-T P.804 subjective evaluation and pick the winners based on the results. The challenge metric is an average of OVRL, SIG, and WAcc (Word Accuracy Rate). See for additional Contest details. 1% \u2013 The Entry was described in an accepted ICASSP 2024 Grand Challenge paper. Winners will be selected within 7 days following the event. Winners will be notified within 7 days following the Event. In the event of a tie between any eligible entries, an additional judge will break the tie based on the judging criteria described above. The decisions of the judges are final and binding. If public vote determines winners, it is prohibited for any person to obtain votes by any fraudulent or inappropriate means, including offering prizes or other inducements in exchange for votes, automated programs or fraudulent ID\u2019s. Microsoft will void any questionable votes.<\/p>\n\n\n\n
Odds<\/h3>\n\n\n\n
The odds of winning are based on the number and quality of eligible entries received.<\/p>\n\n\n\n
General conditions and release of liability<\/h3>\n\n\n\n
To the extent allowed by law, by entering you agree to release and hold harmless Microsoft and its respective parents, partners, subsidiaries, affiliates, employees, and agents from any and all liability or any injury, loss, or damage of any kind arising in connection with this. All local laws apply. The decisions of Microsoft are final and binding. We reserve the right to cancel, change, or suspend this Contest for any reason, including cheating, technology failure, catastrophe, war, or any other unforeseen or unexpected event that affects the integrity of this Contest, whether human or mechanical. If the integrity of the Contest cannot be restored, we may select winners from among all eligible entries received before we had to cancel, change or suspend the Contest. If you attempt or we have strong reason to believe that you have compromised the integrity or the legitimate operation of this Contest by cheating, hacking, creating a bot or other automated program, or by committing fraud in any way, we may seek damages from you to the full extent of the law and you may be banned from participation in future Microsoft promotions.<\/p>\n\n\n\n
Governing law<\/h3>\n\n\n\n
This Contest will be governed by the laws of the State of Washington, and you consent to the exclusive jurisdiction and venue of the courts of the State of Washington for any disputes arising out of this Contest.<\/p>\n\n\n\n
Privacy<\/h3>\n\n\n\n
At Microsoft, we are committed to protecting your privacy. Microsoft uses the information you provide on this form to notify you of important information about our products, upgrades and enhancements, and to send you information about other Microsoft products and services. Microsoft will not share the information you provide with third parties without your permission except where necessary to complete the services or transactions you have requested, or as required by law. Microsoft is committed to protecting the security of your personal information. We use a variety of security technologies and procedures to help protect your personal information from unauthorized access, use, or disclosure. Your personal information is never shared outside the company without your permission, except under conditions explained above. If you believe that Microsoft has not adhered to this statement, please contact Microsoft by sending an email to privrc@microsoft.com<\/a> or postal mail to Microsoft Privacy Response Center, Microsoft Corporation, One Microsoft Way, Redmond, WA 98052<\/p>\n\n\n\n\n\n
Program timeline<\/h2>\n\n\n\n
This challenge is to benchmark the performance of real-time algorithms with a real (not simulated) test set. Participants will evaluate their acoustic echo canceller on a test set and submit the results (audio clips) for evaluation. The requirements for each acoustic echo canceller used for submission are:<\/p>\n\n\n\n
- \n
- October 15, 2023: Release of the datasets and scripts for training and testing<\/li>\n\n\n\n
- December 5, 2023: Blind test set released to participants.<\/li>\n\n\n\n
- December 10, 2023: Deadline for participants to submit their results<\/li>\n\n\n\n
- December 17, 2023: Subjective and objective results released<\/li>\n\n\n\n
- January 2, 2024: Grand Challenge 2-page Papers Due (by invitation only)<\/li>\n\n\n\n
- January 16, 2024: Grand Challenge 2-page Paper Acceptance Notification<\/li>\n\n\n\n
- January 23, 2024: Camera-ready Grand Challenge 2-page Papers Due<\/li>\n\n\n\n
- June 19, 2024: Grand Challenge OJ-SP Papers Due (by invitation only)<\/li>\n<\/ul>\n\n\n\n\n\n
Organizers<\/h2>\n\n\n\n
- \n
- Ross Cutler<\/a>, Microsoft, USA<\/li>\n\n\n\n
- Nicolae-Catalin Ristea, Microsoft, Romania<\/li>\n\n\n\n
- Ando Saabas, Microsoft, Estonia<\/li>\n\n\n\n
- Babak Naderi<\/a>, Microsoft, Germany <\/li>\n\n\n\n
- Sebastian Braun<\/a>, Microsoft Research, USA<\/li>\n\n\n\n
- Solomiya Branets<\/a>, Microsoft, USA<\/li>\n<\/ul>\n\n\n\n\n\n
Related links<\/h2>\n\n\n\n
- \n
- Dataset and test set (opens in new tab)<\/span><\/a><\/li>\n\n\n\n
- Microsoft Conference Management Toolkit (opens in new tab)<\/span><\/a><\/li>\n\n\n\n
- Sample clips representing performance of participating models<\/a><\/li>\n<\/ul>\n\n\n\n
Other challenges<\/h3>\n\n\n\n
- \n
- Speech Signal Improvement Challenge \u2013 ICASSP 2023<\/a><\/li>\n\n\n\n
- Acoustic Echo Cancellation Challenge \u2013 ICASSP 2023<\/a><\/li>\n\n\n\n
- Audio Deep Packet Loss Concealment Challenge \u2013 INTERSPEECH 2022<\/a><\/li>\n\n\n\n
- Deep Noise Suppression Challenge \u2013 ICASSP 2022<\/a><\/li>\n\n\n\n
- Acoustic Echo Cancellation Challenge \u2013 ICASSP 2022<\/a><\/li>\n\n\n\n
- Acoustic Echo Cancellation Challenge \u2013 ICASSP 2021<\/a><\/li>\n\n\n\n
- Deep Noise Suppression Challenge \u2013 ICASSP 2021<\/a><\/li>\n\n\n\n
- Acoustic Echo Cancellation Challenge \u2013 INTERSPEECH 2021<\/a><\/li>\n\n\n\n
- Deep Noise Suppression Challenge \u2013 INTERSPEECH 2020<\/a><\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n\n\n
Real-time track<\/h2>\n\n\n\n
![\"Speech](\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/09\/sig2024_rt-658072cfb6038-1024x346.png\")
<\/figure>\n\n\n\nNon-real-time track<\/h2>\n\n\n\n
![\"Speech](\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/09\/sig2024_nrt-658072c3deb23-1024x297.png\")
<\/figure>\n\n\n\nSample clips<\/h2>\n\n\n\n
- Acoustic Echo Cancellation Challenge \u2013 ICASSP 2023<\/a><\/li>\n\n\n\n
- Microsoft Conference Management Toolkit (opens in new tab)<\/span><\/a><\/li>\n\n\n\n
- Ross Cutler<\/a>, Microsoft, USA<\/li>\n\n\n\n
- For real-time track, the processing must take less than the stride time T_s<\/em> (in ms) to process a frame of size T<\/em> (in ms) on an Intel Core i5 quad-core machine clocked at 2.4 GHz or equivalent processors. For example, T_s = T\/2<\/em> for 50% overlap between frames.<\/li>\n\n\n\n
- Compress the enhanced results files to a single lastname<\/em>.zip file, retaining the same folder and file names as the blind test set (max file size: 1.8 GB).<\/li>\n\n\n\n