# Introduction The Sound blocks are undoubtedly very cool. But wait, you can only play specific sounds you have put before? That's pretty boring. What if you can play any sound, stored in something like a list? This demo shows what may be possible on Scratch in terms of sound reproduction. The music that this project is creating is made solely by a list (which is freqs). It produces a recognizable sound (although with a lot of artifacts). # How it works ## The limitations As we have discussed earlier, we can't put custom sounds on Scratch, so instead we need to somehow recreate sounds from sounds. You might have seen those MIDI players on Scratch that plays music from Scratch's Music extension, however those usually aren't enough to recreate arbitrary sounds. This demo uses something similar, however instead of instruments, it uses sine waves to reconstruct the sound. A sine wave (or a pure tone) is the simplest construct of sound. You may have heard it before—the beep when a speech is censored is a 1 kHz sine wave tone. It is as simple as it sounds, with each tone occupying just a single frequency. With such a simple sound, you may wonder how will you convert normal sounds into them. ## Fourier Transform In the early 19th century (that’s 18xx), this guy called Joseph Fourier found something that turns a function of time into a function of frequency (and phase for those pedantic people). This way, we can turn something like an audio signal into their frequencies (or, in this context, the pitch). Fourier transform (and it's derivatives) are used everywhere. Physics, audio processing, image compression, etc. For this demo, we will use a tool that converts audio into a spectrogram, which is a diagram of frequency over time. ## The ARSS The Analysis & Resynthesis Sound Spectrograph is a really ancient tool that generates spectrograms from audio. I use the ARSS since it allows analysis of frequency on a log scale, while other tools like Scipy can't, at least directly. The log scale is important because while ARSS does return the amplitude of the frequencies, it doesn't return the phases, which when used with linear scale, will result in a dreadful to listen audio. Besides, I don't think Scratch can recreate the phases anyway. I conjecture that this doesn't happen on a log scale since on there the ratio between frequencies are consistent, while linear scales do create inconsistent frequency ratios, including some odd and dissonant ones like 23:24 and 56:33. (Future Gilbert here: It turns out this type of transform is called CQT (constant Q transform). Look it up on Wikipedia to learn more.) The cool thing about the ARSS is that it can also try to reconstruct audio from spectrograms as well. Remember, the ARSS doesn't output the phases of the transform, so it also generates audio from sine waves. And it does that... as good as this demo does. ## Conversion The ARSS outputs an image, which I turn into ASCII text using a custom Python script. I set the ARSS parameters to generate a spectrogram on frequency range 50 Hz to 12800 Hz, with 16 bars per octave and 30 pixels per second. I also set the gamma to 1 (meaning that the amplitude of the song is linearly proportional to the brightness of the image) and the log base of 2. This resulted in an image with a height of 129 pixels. I trim the first column of pixels, since it's not really needed. ## Playback To play back the sound, I use 128 clones to play the tones. While Scratch can't overlay the same sound, it can overlay multiple sound, even on the same sprite. The clones adjust the volume of the tones according to the `current` variable, which represents the current frequencies at this time. (I'm glad audio volumes aren't global.) Currently this is set using a timer, however with some tweaks it can be seeked to a specific point in time. # Results The results of the synthesized sound contains a lot of artifacts. This may be because the spectrogram has a low resolution; about 128 tones. The gamma parameter may also have a role here; it limits the dynamic range of the sound.
