Re: Streaming video over serial?

Joshua Bell wrote:
Michael J. Mahon wrote:

Since the time to transfer a frame is certainly known as soon
as it is captured (even if it is, say, run-length compressed),
the frame rate can be made adaptive to the content. In other
words, if it can run faster, it does.


Yep. In this scenario, since it's doing realtime, maintaining a consistent frame rate isn't necessary - "as fast as possible" is what's desired. The closest analogy would be the "remote desktop" applications like VNC, which attempt to give you a generic mechanism for the local display (and optionally interactivity) of a remote computer's screen (or app window).


Compression here would also be nice, but
decompression on the Apple might be a killer.

Not if it saves more data transfer time than it costs in processor
time. That's the tradeoff to examine, and it, of course, depends
critically on the actual serial data transfer rate. At 115kbps, it
may be necessary to forego decompression just to maintain the UART
transfer rate. For example, interrupt processing of UART data is
not the way to go at this rate--just simple polling and data storage.


Back-of-the-envelope for 115kbps gives me about a 2Hz update rate for hi-res graphics without compression. Given that, I'd probably skip lo-res entirely since that rate is "good enough" for what I'm imagining.

Well, you'll need at least 9 bit times per byte, and that puts you
at 12,800 bytes/sec, or about 1.5 seconds per hi-res screen. That's
actually pretty slow.

Run-length compression could save you some, depending on whether you
compressed the actual screen or an XOR with the previous screen, and,
of course, on how much of the screen is changing each frame.

Compression would have to be *very* simple to fit in the <80 cycles
you have per byte, but since run-length uses character pairs, it might
work out well.

You're right about compression. However, in the particular case I'm thinking about, the graphics will not be amenable to the sort of decompression the Apple could do in realtime, at least for intra-frame. Interframe compression a la MPEG would be feasible to get higher than 2Hz (if less than a full frame changes, transmit only the rectangle that does), but since it's an interactive first-person immersive 3D environment the cases where higher than 2Hz refresh are compelling are when the viewpoint is changing rapidly (i.e you're trying to naviate), which are the worst-case scenarios for intra-frame compression.

(That's not to say that either the more complex intermixed lo-res/hi-res or intra-frame compression wouldn't be useful in more general applications of this notion, i.e. streaming arbitrary video. I just know enough about the data in this case to shelve those ideas for now.)

Give up on anything more complicated than run-length compression of data
or data differences. 80 cycles per byte is a harsh mistress. ;-)

There will be time in the loop to detect a simple protocol escape...


I confess to not knowing enough about serial transmission to know how much putting a handshake between frames would slow things down, but I have to assume "not much". If we're transmitting a frame in 0.5 seconds, spending 0.01 second between frames for the client to say "user pressed 'W' key" is feasible, so doing this within the frame loop is not necessary.

You don't want a handshake, since the only recovery is to keep going!

If the sending machine is sent data, it simply acts on it without any
housekeeping handshakes. All "service" should simply be "best effort".

I'm also not sure how the sender knows when when the receiver is done pulling bits out of a buffer; to reduce perceived latency in this realtime client/server app, the server should capture the frame to send as close to when the client is done receiving the previous frame as possible. That's probably serial communication 101, tho.

The receiver doesn't "pull" bits, it is "sent" to the receiver by
the sender without handshaking.

Since the sender knows exactly when the end-of-frame 2-byte sequencs
has been sent, it knows exactly when the receiver has displayed it.

When you decide to switch to hi-res, you'll be filling the hi-res
buffer while continuing to display the last full lo-res buffer, I
presume. So if you don't finish transferring the full hi-res
screen, the partial frame transfer would never be visible.


Yep. (Again, it seems that at 115kbps I'd just skip lo-res which simplifies things)

You'd be surprised how much better a 16-color, 6 FPS display looks
than a 6-color 0.67 FPS display! (If you want 16-color hi-res, that's
double hi-res, and requires twice as long to send.)

At 6 FPS you can see motion pretty well, but at under 1 FPS it's
pretty hard unless things are changing quite slowly.

It's also quite easy to change, since a simple "switched" command
handler on the Apple II side can be told exactly which screen to
fill next and which to display now. There are really no smarts on
the Apple side at all--in fact, it doesn't even keep count of bytes
transferred, it just stores them in increasing addresses until told
to switch buffers and screens.

For highest speed, the protocol will need to be pretty "fragile",
with recovery from a data transmission error essentially awaiting
the next re-syncing event


Agreed. Again, for this scenario, that's fine.

It's looking like the simplest client implementation is basically:
* initialize the UART to 115kbps
* start a loop -
* read from UART until a "start of frame" signature is seen
* start filling the non-visible hires page
* after 8192 bytes, flip hires pages
* jump to start of loop (i.e. wait for a signature)

Actually, switching buffers and screens is fast enough that you can
probably do it between the end-of-frame command and the next character,
so no "wait for start of frame" is needed.

If you want to accept other, longer to process commands (like play a
sound on the Apple speaker), then the sending machine would anticipate
the delay and insert an appropriate number of in-line nulls and finish
(at just after the time when the Apple would be done in the worst case)
with a "resume data" byte that would send control back into the
"receive data" loop. This could even happen inside a frame.

The next thing to implement would be a simple 1-byte sync signal every 256 bytes; if not seen, assume there was a glitch somewhere and abort the display of this frame, waiting for the next start-of-frame signature.

No, just run open-loop. There's nothing the sender can do to improve
things in case of an error--this is real-time, after all--and you can
easily verify correct or incorrect operation by observing the screen.

I've never done serial programming on either side, but I'm assuming the initialization is trivial and you've provided the guts of the loop already. (Thanks!)

The hard part now seems like it's actually on the sending side, where we need to tackle algorithms for generating decent color hi-res screens. I'd probably start by assuming it's a 140x192 with a fixed 6 color palette, do an error-diffusion dither, and ignore the artifacts from trying to put blue and green within (etc) a byte. Which would make Rich happy, I'm assuming.

I agree completely. Palette reduction is non-trivial (maybe with an
animation, you can get some advantage from the fact that many colors do
not change from frame to frame).

Don't ignore the "color set" bit, or what you produce will be suitable
only for monochrome viewing. Trust me, with only 40 bytes across the
screen, changing random byte's color sets makes a very visible mess.

Maybe a good way to start would be to make a monochrome version. The
conversion is much easier to write. ;-)

BTW, although as David points out, the IIc keyboard can generate
interrupts, I would very much recommend *against* using interrupts.
They will make your character receive loop non-deterministic and
random failures will occur. Further, if you skip keyboard (or any
other) interrupts, *all* Apple II's will run the code perfectly.

-michael

NadaPong: Network game demo for Apple II computers!
Home page: http://members.aol.com/MJMahon/

"The wastebasket is our most important design
tool--and it's seriously underused."
.

Relevant Pages

  • Re: Streaming video over serial?
    ... In my specific scenario, where I'm not actually streaming video but an interactive application, quality actually trumps frame rate at about the 1Hz rate. ... I agree that for actual *usability* from an Apple client, dropping to lores would be great, but it's not necessary in this case. ... the Apple side it goes straight to frame buffer. ... to switch buffers and screens. ...
    (comp.sys.apple2.programmer)
  • Re: Mini-DV vs Analog to the Desktop - Question...
    ... and Digital 8 camcorders record 29.97 frames per second, just like analog ... there is absolutely no difference in frame rate between analog ... You claim DV is better than MPEG2 ... By it's nature, it entails compression of the image at, ...
    (rec.video.desktop)
  • Re: video capture
    ... DVDs are substantially more compressed than ... but DV is a different style of compression than what is ... that each frame is a complete image. ... Capturing baseband video uncompressed requires special ...
    (rec.video.production)
  • Re: DV: digital vs. analog dubs
    ... Look, you can claim all you like the data dropouts don't happen on DV, ... where the DV deck can't mask it and drops in a frame of silence ... Decks may do this kind of correction, ... In interframe compression one of the main ways of saving space is to reduce ...
    (rec.video.production)
  • Re: Streaming video over serial?
    ... frame rate isn't necessary - "as fast as possible" is what's desired. ... You're right about compression. ... I confess to not knowing enough about serial transmission to know how much ... so doing this within the frame loop is not necessary. ...
    (comp.sys.apple2.programmer)
Loading