A MIDI keyboard tells the app exactly which key you pressed, at what moment, and how hard. A microphone has to work all of that out from the sound in the room, and working it out is much harder than it sounds. Both approaches are real and both ship in products people like. The difference is that one of them is a measurement and the other is an estimate, and for feedback that arrives while you are still playing, that difference decides what the software can honestly tell you.
I build SightReader, so I have an obvious stake in this. What follows is the reasoning behind a design decision rather than a case for the decision, and where the trade-off goes against us I have said so.
What MIDI actually transmits
MIDI is not audio. It carries no sound at all. When you press middle C on a MIDI keyboard, the instrument puts three bytes on the wire:
0x90 0x3C 0x54The first byte says “note on, channel 1”. The second is the note number — 60, middle C. The third is velocity, how hard you struck it, on a scale of 1 to 127. When you let go, a matching note-off arrives. The browser timestamps each message on arrival, and that is the whole chain. Nothing in it is inferred. There is no model, no threshold, no confidence score. The keyboard is not describing a sound it made; it is reporting a switch closing.
That is why there is no accuracy figure for MIDI note detection. The question does not apply. If the app says you played an F sharp when you meant to play an F, you played an F sharp.
What a microphone has to do instead
A microphone gets one number many thousands of times a second: air pressure. Everything else has to be reconstructed.
The task has a name — polyphonic automatic music transcription — and it is one of the properly hard problems in audio. A single piano note is not a single frequency. Strike an A at 440Hz and you also get energy at roughly 880, 1320, 1760 and upwards, decaying at different rates, shaped by the instrument and the room. That is the harmonic series, and it is what makes a piano sound like a piano.
Now play a chord. The partials of the lower notes land on or near the fundamentals of the higher ones. An A and the E above it share a great deal of spectral content. The algorithm has to decide whether the energy at a given frequency is a note somebody played or an overtone of a note somebody played, and it has to do that for every note in the chord, continuously, while the previous notes are still ringing and the sustain pedal is possibly holding all of them.
It is a genuinely impressive thing that this works at all. It does work.
What the research actually achieves
The current state of the art for doing this in real time on a phone is Mobile-AMT, presented at EUSIPCO 2024 by Yuta Kusaka and Akira Maezawa at Yamaha. It is good work and the numbers are strong:
- 96.30% note F1 on clean MAESTRO recordings.
- 93.15% note F1 on real-world recordings, which is the number that matters, and it is much better than it has any right to be.
- It runs in real time on mobile hardware, cutting computational cost by roughly 83% against comparable methods.
Two other figures from the same body of work matter more for what I do:
- Note-with-offset F1 falls to 76.80% even on clean studio audio. Offsets are note endings — durations. So the weakest number in the literature is precisely the one rhythm assessment depends on.
- Trained without heavy augmentation for room acoustics, background noise and device characteristics, real-world F1 collapses to 78.84%. Robustness in the wild is not free; it has to be engineered in, and a product that has not done that work will be much worse than the headline suggests.
A 2024 evaluation paper, Towards Musically Informed Evaluation of Piano Transcription Models, makes a further point worth taking seriously: high F1 scores can mask musical failures. Onset timing degrades measurably as noise levels rise, and there is a statistically significant drop between clean benchmark datasets and the same material re-recorded acoustically. The score can look fine while the musical result is not.
The latency arithmetic
Mobile-AMT reports 174ms of latency. Over MIDI, the round trip from key press to a coloured notehead on screen is under 20ms in a browser — the Web MIDI API delivers the message with a hardware timestamp and the rest is rendering.
174 against 20 is roughly nine times slower, but the ratio is not the point. Here is the sum that is:
At 60 beats per minute, one beat is 1000ms. So 174ms is about a third of a beat. At 120bpm — a brisk Grade 3 piece — it is two thirds of a beat.
Think about what that means for feedback while you play. You strike a note. By the time the app has decided what it was, you are most of the way to the next one. You cannot colour a notehead as it happens, you cannot tell a player they are dragging while they are dragging, and you certainly cannot score onset timing to a tolerance finer than the delay in measuring it. Our rhythm tolerance is 120ms at the tightest. That budget is gone before the transcription finishes.
This is why I keep coming back to latency rather than accuracy as the deciding argument. Accuracy you can improve with a better model. The latency is doing real work — it is the time the algorithm needs to hear enough of the note to be sure what it was — and shrinking it below the length of a piano attack is not a matter of engineering effort.
The error arithmetic
Now the accuracy side, because the way the number behaves over a piece is worse than the number suggests.
Take 95% note accuracy, comfortably inside what the research reports for real-world audio. A Grade 5 piece runs to roughly 500 notes. Five per cent of 500 is about 25 wrong verdicts per attempt.
Twenty-five is not a disaster in itself. The problem is which twenty-five. You do not know. There is no marker on screen that says “this one was the algorithm’s fault”, because if the algorithm knew it was wrong it would not have been wrong. So a player looking at a red notehead has to decide whether they misread the score or the software misheard the room, and there is no way to tell from the inside.
For a product whose entire value is that you can trust the verdict, an untraceable error rate is corrosive in a way a slower verdict is not. It teaches you to discount the feedback, and feedback you discount is worth nothing. MIDI has no equivalent failure because it is not making a judgement.
You do not have to take my word on the practical consequences either. Note-detection failure is the most common complaint in reviews of the two most heavily marketed microphone-based products. Simply Piano’s own help centre is direct about the remedy: connecting a MIDI keyboard “will give perfect note recognition and will solve any issues with note recognition, particularly when playing more advanced music”. The rest of that page is about room noise, phone cases over the microphone, and playing louder. The research explains why those instructions exist.
Why offsets matter most for us
Sight reading has two halves: the right notes and the right time. SightReader’s Pitch & Rhythm mode grades the second half independently — with the metronome running, every onset is measured against the beat, and the results tell you what percentage landed on time, early and late, plus your average tendency in milliseconds.
That whole feature runs on MIDI timestamps, and the offset number above is why. Rhythm assessment is a measurement of when things start and stop. It is the one job where the literature’s weakest figure is the figure you depend on, and where a 174ms delay is not a rounding error but a third of a beat.
If I could not measure timing to a tolerance I trusted, I would not ship a timing score. Shipping one anyway, on inferred data, would produce a number that looks authoritative and is not, which is the worst outcome available.
Why we don’t import MIDI files either
This is the part that surprises people, because it looks like a limitation with no argument behind it. SightReader accepts any MusicXML file and no other format. Not MIDI, not audio, not PDF.
It is the same decision as the microphone one, pointed the other way.
MIDI carries performance data. It is superb at that. What it does not carry is notation. A MIDI file has no clef assignment, so the software has to guess which hand plays what. It has no key-signature semantics, so it cannot tell you whether that black key is an A flat or a G sharp — the two are the same MIDI note number 68 and completely different things on a page. It has no voicing, so overlapping lines in one hand collapse into a soup of simultaneous notes. It has no beaming, no articulation, no rests as written, no spelling of any kind.
Rendering a readable score from raw MIDI means inferring every one of those. That is the same class of problem as transcribing audio, just milder — and it produces the same kind of output, which is a plausible guess. If you have ever imported a MIDI file into notation software and watched it produce a bar of quintuple-dotted rests, you have seen it.
MusicXML already contains the notation. Clefs, key, spelling, voices, beams, slurs, fingering, the lot. What you see on screen is what the composer or editor actually wrote, and when the app marks a note wrong it is marking it against the score rather than against a reconstruction of the score.
If your music is in MuseScore, Sibelius, Finale or Dorico, exporting MusicXML takes about twenty seconds. I have written a short guide to doing it in each.
The principle, stated once
We only score what we know exactly.
MIDI in from the keyboard rather than a microphone. MusicXML in for scores rather than inferred notation. It is the same decision twice, and it is the reason the feedback can be trusted: at no point in the chain does the software guess at something and then mark you against its guess.
Everything above is evidence for that one sentence. It is also, if I am honest about it, a constraint I chose — it rules out a market, and the next section is about who is in it.
When a microphone app is the better choice
If you have an acoustic piano and no MIDI keyboard, none of the above helps you. You cannot use SightReader’s core loop at all, and a microphone app is not a compromise for you — it is the only thing that works.
That is not a small group. It is uprights in family homes, school practice rooms, and every pianist who has a perfectly good instrument and no interest in buying a second one with a USB socket. For them, Sight Reading Factory is a well-made and well-regarded tool used by a lot of teachers, its microphone assessment is its primary method rather than a fallback, and it covers more than 60 instruments including voice. We do piano, over MIDI, and that is all we do.
Simply Piano, Skoove and flowkey all accept microphone input too, alongside MIDI, and all three are polished products with catalogues we cannot match.
The trade-off is real in both directions. An estimate you can use beats a measurement you cannot.
What we offer without a MIDI keyboard
Two things, and I would rather undersell them than have you arrive expecting more.
There is an on-screen piano you can play with your computer keyboard. It covers two octaves, C3 to B4, which is enough for Initial Grade and early Grade 1 pieces and enough to see whether you like the app. It is not enough for real repertoire and it is nothing like playing the piano.
There is also honour mode: play the piece on whatever instrument you have, then tell the app how it went. That keeps your practice history intact. It is self-reported, so it is exactly as accurate as you are.
Neither is a substitute for the thing this article is about. If you have an acoustic piano and no plans to get a MIDI keyboard, use one of the microphone apps.
MIDI and microphone, side by side
| App | What reaches the app | Latency | Note error rate | Works with an acoustic piano | Degrades in a noisy room |
|---|---|---|---|---|---|
| Microphone | Audio, transcribed | ~174ms | ~4–7% on real-world audio | ✓ | ✓ |
| MIDI | Note number, velocity, timestamp | Under 20ms | None — not inference | ✗ | ✗ |
FAQs
Do I need a MIDI keyboard for a sight-reading app?
For note-by-note feedback while you play, yes — or you need an app that listens through a microphone and accepts the error rate that comes with it. MIDI is a data protocol: the keyboard reports which key went down, when, and how hard, with no guessing involved. Apps that listen through a microphone have to infer all of that from sound, which is slower and imperfect. Several good apps do it anyway, and if you have an acoustic piano it is your only option.
Can an app hear my acoustic piano?
Yes. Sight Reading Factory, Simply Piano, Skoove, flowkey and others accept microphone input, and they work. What they cannot do is match the timing precision or the certainty of MIDI: the best published real-time piano transcription runs at about 174ms of latency and mis-labels a few notes in every hundred, and neither number is visible to you while you play. If you have an upright and no MIDI keyboard, a microphone app is the right choice and SightReader is not.
Why can’t apps detect chords from audio?
They can. Polyphonic piano transcription works and works well — recent research reports note F1 scores in the mid-90s on real-world recordings. The problem is not that it fails, it is that it is approximate and slow in a way that matters for live feedback. A single piano note produces a whole harmonic series, and the partials of one note land on the fundamentals of others, so separating a chord takes inference. Inference takes time and produces occasional wrong answers.
What’s the difference between MIDI and MusicXML?
MIDI describes a performance: which note, when it started, when it stopped, how hard. MusicXML describes notation: clefs, key signatures, enharmonic spelling, voices, beams, articulation. You can generate a rough MIDI file from a score, but going the other way means guessing at everything the MIDI file does not carry. That is why SightReader takes MIDI in from your keyboard and MusicXML in for scores.
Does SightReader work without a MIDI keyboard?
Partly. There is an on-screen piano you can play with your laptop keyboard, covering two octaves (C3–B4), which is enough for beginner pieces and fine for trying the app out. There is also honour mode, where you play on any instrument and report your own result. Neither gives you note-by-note feedback on a real piano, and neither is a substitute for a MIDI keyboard.
Sources
- Kusaka, Y. and Maezawa, A., Mobile-AMT: Real-Time Polyphonic Piano Transcription for In-the-Wild Recordings, EUSIPCO 2024 — PDF
- Towards Musically Informed Evaluation of Piano Transcription Models, 2024 — arXiv:2406.08454
- Kong, Q. et al., High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times, 2020 — arXiv:2010.01815
- Simply Piano Help Center, Ways to Improve Note Recognition in Simply Piano — help.hellosimply.com
Figures checked 1 August 2026.
