Audio

Audio support is built upon the SoundCard library. The provided API gives a simple asyncio-based wrapper of the library, which integrates directly with other components of rtcbot.

The library is made up of two objects: a Speaker and Microphone. The Microphone gathers audio at 48000 samples per second, and gives the data in chunks of 1024 samples. The data is returned as a numpy array of shape (samples,channels).

The speaker performs the reverse operation: it is given numpy arrays containing audio samples, and it plays them on the computer’s default audio output.

Basic Example

With the following code, you can listen to yourself. Make sure to wear headphones, so you don’t get feedback:

import asyncio
from rtcbot import Microphone, Speaker

microphone = Microphone()
speaker = Speaker()

speaker.putSubscription(microphone)

try:
    asyncio.get_event_loop().run_forever()
finally:
    microphone.close()
    speaker.close()

Naturally, the raw data can be manipulated with numpy. For example, the following code makes the output five times as loud:

import asyncio
from rtcbot import Microphone, Speaker

microphone = Microphone()
speaker = Speaker()

@microphone.subscribe
def onData(data):
    data = data * 5
    if speaker.ready:
        speaker.put_nowait(data)

try:
    asyncio.get_event_loop().run_forever()
finally:
    microphone.close()
    speaker.close()

By checking if the speaker is ready, we don’t queue up audio while it is initializing (if the microphone starts returning data before the speaker is prepared). This allows us to hear the audio with low latency. This effect was automatic in the first example, because a subscription to microphone was not created until microphone.get was called by the speaker.

Warning

This is one of the fundamental differences between video and audio in RTCBot - dropping a video frame is not a big deal, so the cameras automatically always return the most recent frame. However, dropping audio results in weird audio glitches. To avoid this, audio is queued. This means that a subscription that is not actively being read will keep queueing up data indefinitely. Make sure to unsubscribe the moment you stop using an audio subscription, or your code will eventually run out of memory!

API

class rtcbot.audio.Microphone(samplerate=48000, channels=None, blocksize=1024, device=None, loop=None)

Bases: ThreadedSubscriptionProducer

Reads microphone data, and writes audio output. This class allows you to output sound while reading it.

Parameters

• samplerate (int,optional) – The sampling rate in Hz. Default is 48000.

• channels (int,list(int),optional) – The index of channel to record. Allows a list of indices. Records on all available channels by default.

• blocksize (int,optional) – Records this many samples at a time. A lower block size will give lower latency, but higher CPU usage.

• device (soundcard._Microphone) – The soundcard device to record from. Uses default if not specified.

close()

Shuts down data gathering, and closes all subscriptions. Note that it is not recommended to call this in an async function, since it waits until the background thread joins.

The object is meant to be used as a singleton, which is initialized at the start of your code, and is closed when exiting the program.

property closed

Returns whether the object was closed. This includes both thrown exceptions, and clean exits.

property error

If there is an error that causes the underlying process to crash, this property will hold the actual Exception that was thrown:

if myobject.error is not None:
    print("Oh no! There was an error:",myobject.error)

This property is offered for convenience, but usually, you will want to subscribe to the error by using onError(), which will notify your app when the issue happens.

Note

If the error is not None, the object is considered crashed, and no longer processing data.

async get()

Behaves similarly to subscribe().get(). On the first call, creates a default subscription, and all subsequent calls to get() use that subscription.

If unsubscribe() is called, the subscription is deleted, so a subsequent call to get() will create a new one:

data = await myobj.get() # Creates subscription on first call
data = await myobj.get() # Same subscription
myobj.unsubscribe()
data2 = await myobj.get() # A new subscription

The above code is equivalent to the following:

defaultSubscription = myobj.subscribe()
data = await defaultSubscription.get()
data = await defaultSubscription.get()
myobj.unsubscribe(defaultSubscription)
newDefaultSubscription = myobj.subscribe()
data = await newDefaultSubscription.get()

onClose(subscription=None)

This is mainly useful for connections - they can be closed remotely. This allows handling the close event.

@myobj.onClose
def closeCallback():
    print("Closed!)

Be aware that this is equivalent to explicitly awaiting the object:

await myobj

onError(subscription=None)

Since most data processing happens in the background, the object might encounter an error, and the data processing might crash. If there is a crash, the object is considered dead, and no longer gathering data.

To catch these errors, when an unhandled exception happens, the error event is fired, with the associated Exception. This function allows you to subscribe to these events:

@myobj.onError
def error_happened(err):
    print("Crap, stuff just crashed: ",err)

The onError() function behaves in the same way as a subscribe(), which means that you can pass it a coroutine, or even directly await it:

err = await myobj.onError()

onReady(subscription=None)

Creating the class does not mean that the object is ready to process data. When created, the object starts an initialization procedure in the background, and once this procedure is complete, and any spawned background workers are ready to process data, it fires a ready event.

This function allows you to listen for this event:

@myobj.onReady
def readyCallback():
    print("Ready!)

The function works in exactly the same way as a subscribe(), meaning that you can pass it a coroutine, or even await it directly:

await myobj.onReady()

Note

The object will automatically handle any subscriptions or inserts that happen while it is initializing, so you generally don’t need to worry about the ready event, unless you need exact control.

property ready

This is True when the class has been fully initialized, and is ready to process data:

if not myobject.ready:
    print("Not ready to process data")

This property is offered for convenience, but if you want to be notifed when ready to process data, you will want to use the onReady() function, which will allow you to set up a callback/coroutine to wait until initialized.

Note

You usually don’t need to check the ready state, since all functions for getting/putting data will work even if the class is still starting up in the background.

subscribe(subscription=None)

Allows subscribing to new data as it comes in, returning a subscription (see Subscriptions):

s = myobj.subscribe()
while True:
    data = await s.get()
    print(data)

There can be multiple subscriptions active at the same time, each of which get identical data. Each call to subscribe() returns a new, independent subscription:

s1 = myobj.subscribe()
s2 = myobj.subscribe()
while True:
    assert await s1.get()== await s2.get()

This function can also be used as a callback:

@myobj.subscribe
def newData(data):
    print("Got data:",data)

If passed an argument, it attempts to use the given callback/coroutine/subscription to notify of incoming data.

Parameters

subscription (optional) –

An optional existing subscription to subscribe to. This can be one of 3 things:

1. An object which has the method put_nowait (see Subscriptions):

   q = asyncio.Queue()
   myobj.subscribe(q)
   while True:
       data = await q.get()
       print(data)
   

2. A callback function - this will be called the moment new data is inserted:

   @myobj.subscribe
   def myfunction(data):
       print(data)
   

3. An coroutine callback - A future of this coroutine is created on each insert:

   @myobj.subscribe
   async def myfunction(data):
       await asyncio.sleep(5)
       print(data)
   

Returns

A subscription. If one was passed in, returns the passed in subscription:

q = asyncio.Queue()
ret = thing.subscribe(q)
assert ret==q

unsubscribe(subscription=None)

Removes the given subscription, so that it no longer gets updated:

subs = myobj.subscribe()
myobj.unsubscribe(subs)

If no argument is given, removes the default subscription created by get(). If none exists, then does nothing.

Parameters

subscription (optional) – Anything that was passed into/returned from subscribe().

unsubscribeAll()

Removes all currently active subscriptions, including the default one if it was intialized.

class rtcbot.audio.Speaker(samplerate=48000, channels=None, blocksize=1024, device=None, loop=None)

Bases: ThreadedSubscriptionConsumer

close()

The object is meant to be used as a singleton, which is initialized at the start of your code, and is closed when exiting the program.

Make sure to run close on exit, since sometimes Python has trouble exiting from multiple threads without having them closed explicitly.

property closed

Returns whether the object was closed. This includes both thrown exceptions, and clean exits.

property error

If there is an error that causes the underlying process to crash, this property will hold the actual Exception that was thrown:

if myobject.error is not None:
    print("Oh no! There was an error:",myobject.error)

This property is offered for convenience, but usually, you will want to subscribe to the error by using onError(), which will notify your app when the issue happens.

Note

If the error is not None, the object is considered crashed, and no longer processing data.

onClose(subscription=None)

This is mainly useful for connections - they can be closed remotely. This allows handling the close event.

@myobj.onClose
def closeCallback():
    print("Closed!)

Be aware that this is equivalent to explicitly awaiting the object:

await myobj

onError(subscription=None)

Since most data processing happens in the background, the object might encounter an error, and the data processing might crash. If there is a crash, the object is considered dead, and no longer gathering data.

To catch these errors, when an unhandled exception happens, the error event is fired, with the associated Exception. This function allows you to subscribe to these events:

@myobj.onError
def error_happened(err):
    print("Crap, stuff just crashed: ",err)

The onError() function behaves in the same way as a subscribe(), which means that you can pass it a coroutine, or even directly await it:

err = await myobj.onError()

onReady(subscription=None)

Creating the class does not mean that the object is ready to process data. When created, the object starts an initialization procedure in the background, and once this procedure is complete, and any spawned background workers are ready to process data, it fires a ready event.

This function allows you to listen for this event:

@myobj.onReady
def readyCallback():
    print("Ready!)

The function works in exactly the same way as a subscribe(), meaning that you can pass it a coroutine, or even await it directly:

await myobj.onReady()

Note

The object will automatically handle any subscriptions or inserts that happen while it is initializing, so you generally don’t need to worry about the ready event, unless you need exact control.

putSubscription(subscription)

Given a subscription, such that await subscription.get() returns successive pieces of data, keeps reading the subscription forever:

q = asyncio.Queue() # an asyncio.Queue has a get() coroutine
myobj.putSubscription(q)

q.put_nowait(data)

Equivalent to doing the following in the background:

while True:
    myobj.put_nowait(await q.get())

You can replace a currently running subscription with a new one at any point in time:

q1 = asyncio.Queue()
myobj.putSubscription(q1)

assert myobj.subscription == q1

q2 = asyncio.Queue()
myobj.putSubscription(q2)

assert myobj.subscription == q2

put_nowait(data)

This function allows you to directly send data to the object, without needing to go through a subscription:

while True:
    data = get_data()
    myobj.put_nowait(data)

The put_nowait() method is the simplest way to process a new chunk of data.

Note

If there is currently an active subscription initialized through putSubscription(), it is immediately stopped, and the object waits only for put_nowait():

myobj.putSubscription(s)
myobj.put_nowait(mydata) # unsubscribes from s

assert myobj.subscription is None

property ready

This is True when the class has been fully initialized, and is ready to process data:

if not myobject.ready:
    print("Not ready to process data")

This property is offered for convenience, but if you want to be notifed when ready to process data, you will want to use the onReady() function, which will allow you to set up a callback/coroutine to wait until initialized.

Note

You usually don’t need to check the ready state, since all functions for getting/putting data will work even if the class is still starting up in the background.

stopSubscription()

Stops reading the current subscription:

q = asyncio.Queue()
myobj.putSubscription(q)

assert myobj.subscription == q

myobj.stopSubscription()

assert myobj.subscription is None

# You can then subscribe again (or put_nowait)
myobj.putSubscription(q)
assert myobj.subscription == q

The object is not affected, other than no longer listening to the subscription, and not processing new data until something is inserted.

property subscription

Returns the currently active subscription:

q = asyncio.Queue()
myobj.putSubscription(q)
assert myobj.subscription == q

myobj.stopSubscription()
assert myobj.subscription is None

myobj.put_nowait(data)
assert myobj.subscription is None