#25 VGA, DSP, and final projects

Bruce Land
2 followers
Oct 10, 2023
4:20 VGA memory use models
15:20 Speech and the MEL spectrum
24:00 available DSP modules
28:00 high quality random numbers and chemical kinetics
more final project examples
Morning folks
So a couple of yeah a week or so ago. I I wrote a
Piazza note asking if anybody had any feeling about lectures for next
semester and of course nobody wrote back because that's a semester away, but
what do you think about the idea of replacing one or more some lectures a week with lab sessions and
The idea would be that in that lab session. There would be a checkpoint for the lab the current lab
so there'd be effectively a
required a
required exercise for the lab period
That that seemed like a reasonable thing to do any opinion on that
Yes
That's that's true that that's that's very true that there there are certain
Kinds of developments that that suggest that you you have everything
Flowing down and nothing works until you all put it together, but it's very hair raising to do that
it's it's I
End up chewing on the wall every every two weeks in this class because of of
It doesn't seem like anything's getting done. It's not getting done suddenly. It's done
Which means that there are a lot of people were up all night for two days or three days
and
I'd like to avoid that if I could
Any what about what about looking at?
Lectures on YouTube and then coming in are people okay with that
Would they actually look at them? No
So that wouldn't work
So there's there is the one good thing about lectures is there's some economy of scale. I get to talk to all of you at
once
half of you at once anyway, and
Without lectures, there's there's no way of getting that economy of scale except to
Write lots of Piazza notes, which then you wouldn't read
Because the more there are the less you read
So I'm not quite sure how to do this, but if you have any thoughts about it, I'd love to hear it
Either now or in in email or whatever as far as this semester goes
I thought I'd talk today about
Some stuff I've already written that could be usable for or useful for final projects
So you don't reinvent the wheel by say rewriting DSP, but I saw people doing last semester in in
4760 and
I'm going to talk a little bit about what the infrastructure that exists and
Then I'll be happy to talk about anything else you want or
Or I'd also be perfectly happy to just go to the lab and we can spend the rest of the semester
Meeting in the lab. What are your thoughts on this?
Lab more talk after today. What's the what's the scoop?
Oh
It's it's hard to invite people and
That would be useful I mean, I'd like to hear from
Zhang about high-level tools for instance, and maybe sooner or later. I will
Probably should take his course and
But I don't think that that's feasible for this semester
I think it's too late for that. But what do you think who has an opinion?
Lab time more lab time less lecture time
okay, so
So then today I want to just talk about some of the the the general
stuff that exists already on the course site and
In particular for the last few days. I've been messing with
Memory models and how to run VGA
Effectively and there's two separate use models
That both make sense under different circumstances
one is
put the VGA
buffer in
SD RAM
by the way, the default model of
University computer puts part of it SD RAM and part of it in in on chip RAM, and I'm not sure why
Hmm so either put it in SD RAM or
Put it in s RAM
the advantage of SD RAM is
It's inexpensive. It's off-chip and
So the real estate is not very
Worth very much s RAM is on chip and the VGA buffer uses about
50
percent of the s RAM on chip
So you're taking a big hit on the on-chip s RAM
On the other hand, this is single threaded and through Q sys the maximum rate
You can write this with VGA still updating is about six million
Transactions per second. So let's call it six million pixels
Per second or if you pack pixels into 32 bits, it might be as high as 30 million or 24 million pixels
Say on chip memory, you can write as fast as your state machine will run so you could easily get
50 to
100
million
Pixels per second written as long as you set up the s RAM to be dual port
So that VGA gets access all the time to one port and
Your state machine gets access all the time to another port, which then don't interact
so
The s RAM solution uses a lot of memory, but it's really fast
SD RAM uses no on-chip memory, but it's
Okay fast not hugely fast
Yeah
Yes, there is there you can write your own off you can write your own SD RAM controller, I believe in the
university
IP there is a
dedicated SD RAM controller and
If you use that you can probably go up again about a factor of 10 in writing speed
Over Q sys, right?
It is however single ported
So you can't do the trick of share of a high high input into one port and high output out of
another port for the VGA
So unable to update. Thank you
so I
Finally documented this stuff. We go down the Avalon bus master peripheral
There's a
section here now that
Uses VGA display it uses a bus master as a GPU
God and
Displays from SD RAM and then another section that does this it uses the same
HPS code with the same function that the FPGA is a GPU for rectangles. It's boring, but it's fast and
The only difference is that
This section is is displaying from SRAM and the top one is displaying from SD RAM
turns out there's a little bit of a tweak and that is that
SD RAM you could you have so much SD RAM. You don't care if you waste a little bit of memory
So you put it in XY mode and X is 10 bits, but you're only using
640 codes out of
1024 so you're losing all over 30% of the memory because you're just you have lots of vacant spot
to use SRAM which is a
Sparse resource you don't want to waste 30%
so you put the
VGA into
sequential address mode and
the only change in the code is that the
Shift by 10 bits to insert Y into the right address becomes a multiply by
640 instead of a shift by 10 bits
the downside is oh
Wait, I can't think of any downside because of because it multiplies the same speed as a shift on the arm architecture more or less
within a cycle or so and
the and
Multiply by a constant on the FPGA does not necessarily instantiate a multiplier
It just uses the right bit waiting to do it and instantiates a bunch of adders
So so why not do that? So so
This mode is this GPU with fast display from SRAM
uses
sequential
addressing which means
Everything has to be changed in the way that the HPS works
and
so I parameterized it made a a
single
point that has to be modified in the HPS code and
that is that
for the
Why can't I zoom this today? Oh different browser?
That sucks. Okay. Let me go back to the Safari browser
That's better
So for packed virtual for packed VGA memory
now the
the
the address the pixel pointer is given by
Y times 640 instead of Y shifted 10 bits. That's the only change but that is the only point in the code
that's that now has to be changed to shift back and forth between
packed and and and
sequent and XY
so
There's two memory models
Use the one that fits your project the best do you need brute speed?
But you don't need much memory for other things or do you need lots of memory?
Some people are interested in doing video input video capture this this charming picture of me sitting at my desk trying to
simultaneously
flip a switch and take a picture
is
Overlaid on graphics being drawn by the HPS so that the disks are being drawn by the HPS
the video is being captured by the
NTSC
television input port which is hardware on the board and
Is being displayed direct as being copied from the input port directly into static RAM?
Where it is then copied into?
SD RAM by
Hardware if you need video input you can just grab
The image from the SRAM and use it
it's just
It's a dual ported memory. You can just reach in and grab that image out and do anything you want with it
So it's using SRAM to
Record the incoming television and SD RAM to output the VGA
70 kilobytes
320 by 240
Under some circumstances you can merge them
Some circumstances you can't
so if I had been smart enough at the time to use a V SRAM VGA display and
Write into that same dual ported memory from the
Television capture I could have done it all on on chip
SRAM
I'm not sure
The reason is that you have a rather a rather high
write rate for the VGA that is competing against a rather a
read rate for the VGA which is competing against a rather high write rate for the
video capture and
There may be artifacts
Although I have not experimented with with with setting the bus arbitration
share to favor the VGA
There's just many there's lots of parameters here to push back and forth
I just haven't had time to do them all so you could you could experiment with that
So there's some VGA cat or some NTSC capture that's already been done
Some optimization for display, but there's some other stuff that I don't know if you had a chance to look at
I've been playing around with DSP a little bit. There's a
One group was interested in doing
some sort of speech analysis and
And
It's a really good idea to read the literature on speech analysis before you try and do it but
one of the central concepts is the
male spectrum
which
uses the psychophysical principle that
Your ears are not linear Fourier transformers, but rather
logarithmically distributed
Frequency analyzers and
therefore if you want to capture the correct kind of information with minimum bandwidth that
Corresponds to what you use
in your head in your ear is you you should build a
set of filters which follow a male scale and
So I I wrote one of these because
Because I
Want to play with it, so this this is a male filter bank. It's got 32 filters
they're each second-order BAM pass filters and
At low frequency you can see that they're linearly spaced
About every hundred Hertz or so and at high frequencies
they are
logarithmically spaced and
They are in fact distributed according to a male distribution
Which you can slurp from the MATLAB code, which is linked up here. It's a it's a fairly funky formula
I got of course from Wikipedia like everybody else does and
then then synthesized into into MATLAB, but you can certainly use that if you
So one thing I've wanted to do is be for particularly for a pitch shifter is you analyze
The voice into those 32 male channels, and then you play back a sine wave
proportional to the amplitude of each
input energy
But you shift the sine wave frequency slightly and you get a shifted output
So this is this is my voice played back through as 32 sine waves
Three four five six seven eight
Nine zero
So that's pretty understandable
But that's a good place to start for capturing what you need to capture
To understand voice you would then take the male
coefficients
you'd
Square them or take the absolute value and low pass filter them
and
Come out with then a
series of energies
32 different energies
profiles for each word
Through each filter, so it'd be the power over time through each filter that gives you a data set
That's fairly low dimensional. It's fairly low rate and
which is
moderately unique for each word
So that might be a piece that you can carry out on the FPGA
Before you send the data over to the HPS for serious analysis. Oh
yes, but there's a
generally audio loopback is
documented here and
I wrote a decimator which allows you to
to
Convert from 48 kilohertz to 8 kilohertz the decimation filter
attenuates the 48 kilohertz signal at 4 kilohertz
Which you have to have for an 8 kilohertz band bandwidth, so you don't get too much aliasing
So that's the slightly ugly filter
Form for the for the decimation filter
then
Then you drop the filter rate to 8 kilohertz all of the computation requirements drop by a factor of 6 and
Building the filters is easier
So decimation filters we are I have running, but in fact if we scroll down slightly
pass the break to the
older pages I
wrote a ton of
filters for
The d2 which are just verilog and so they run on the on the
1 d 1
including 8-bit fixed point 27-bit fixed point
Those are good for audio rate if if you decide you're going to do
real-time filtering of a video signal at 30 mega samples per second
then you need to go to a
parallel pipeline version
Which is runs at bus rate at 50 or 100 megahertz and you can
Systolically pump data through that all day long and at at full bus rated video rate
many of the students in
Past projects have done 2d filters for video
so
Let's say that you want to
Find the edges in a video image
you might
You might take the image
the VGA image and
Put a
small
Convolution
kernel at every point on the image and
For a first derivative the convolution kernel might be
Subtract
all of the pixels on the left from all the pixels on the right and
that'll give you the approximation of the first derivative in the x-direction and
Do the same thing?
for the y-direction
to get an edge detached so to make an edge image, which will give you the outline of say a face or a hand and
And
You can pipeline the image analysis so that this image
the the multiplication by these two
arrays
happens that pixel rate
You make it completely real-time at video at VGA rate
It requires a pipeline that is three lines long so you can get nearest neighbors. Oh
that's only
1,800
Bytes or so that's easy
so there are
Various kinds of of
mostly IRR filters a
Fixed point CIC cascaded integrator comb which is a better form of a decimation filter
Soft radio transmitters
Think twice before you build those they're illegal
Carpless strong string synthesis
so there's some examples of how you use a a
memory block as a shift register, but
Remember that there is a shift register primitive in in
Cordis that allows you to build arbitrary length shift registers
Which is probably what you want to use rather than my hacked together memory block
Also some also some some sections on your direct digital synthesis you've already done noise
Noise synthesis, it's really nice when you're when you have video game to make a sound
Sound so that you can
you can
represent broadband
events
Represented by white noise and in fact for the for the carpless strong algorithm for string synthesis if you want a plucking sound
You have to put energy into the string broadband energy and the best broadband energy
Well, what's broadband energy on on a string? It would be an impulse function on the string
Hard to get a lot of energy into the string that way because if you make this very high your arithmetic will become nonlinear
But broadband energy can also be white noise distributed over the string
It's just the phase is different
The only difference between this signal and that signal if this is white broadband white noise is the phase of the sine waves
Your ear is phase insensitive more or less
so
One way to drive a string is to start it with white noise on it and some adaptive noise cancellation and and some other stuff
So those are already written, I'll be happy to discuss those with whoever
Needs it we've already talked about floating-point
there's a some information on
More information on on
Generation of sounds a
Couple of small CPUs if you're into small CPUs, or you want to hack together a CPU to do some small task
You don't necessarily have to write it yourself
We've already written some of those rs-232 transmitted interface. Let's say you want to send data back from the
FPGA to the PC one way to do it would be serial rs-232 on
This architecture though. My guess is that you would use UDP
much faster
UDP is oh, yes is documented
Where where on the on the University computer page down towards the bottom
there's a
scheme from
Matlab through the arm to the to the FPGA and you go either way on this
UDP is extremely easy to set up because it's just Linux. There's a whole library for supporting
Ethernet UDP you get Ethernet rates pretty handy probably a good idea if you do that though to
Start worrying about interprocess communication
IPC so that you can spawn a thread
That does the UDP
communication, but but does not block other
computation
Other computation in a fit of boredom one summer. I I wrote this thing which solves
Uses rolling dice to solve chemical kinetics
equations
Because after all at some low level
molecules hit and either stick or fall apart and
the hit versus stick fall apart is
May as well be a random process because it depends on lots of degrees of freedom of the orientation of one molecule against another
so you can completely throw away the differential equations and
simulate it by throwing dice for each interaction and
The only and the only trick is to make the dice good enough that you don't start getting
Correlations and the Oregonator is a particular kind of chemical oscillator that was invented in Oregon
so
I was using it as an example
But you can you can the main reason for bringing this up is I wrote a pretty good quality random number generator
based upon a
1970s
Physical review article and it is a hundred and twenty eight bit
127 bit
Linear feedback shift register, so it repeats every two to the hundred and twenty seventh
ticks of the clock
so that's about
every ten to the
Fortieth time ticks and so at at ten to the eighth per second
that's about
Two to the ten to the thirty-two seconds, which is a great deal more than the age of the universe
So effectively this never repeats
But it's folded in such a way that you get a 32-bit output on every two clock cycles
So you can put out a lot of independent numbers if you need a lot of independent random numbers
It passes I haven't done. I haven't done extensive
Testing on my implementation, but it passes kind of the
zeroth order approximation for random numbers
Seems to be uncorrelated
so
There's low. There's low precision random numbers and then high precision random numbers that are already written
And you could see some interesting things if you have just a few molecules
Then the system is dominated by
Poisson Poisson noise and you see these waveforms the black lines of the ODE
solution if
You go to a few thousand molecules
the the statistical output converges to the ODEs as you would hope and
So the black lines overlap the colored lines almost exactly
So that was kind of amusing to do. Oh
And in the process I wrote the the serial transmitter so I could get back data back to MATLAB
This is in the days before we had UDP on the on the processor chip
so
before you do a lot of
Inventing the wheel come talk about if you need some algorithm. It may be that some student has already written it
It may be that I've written it
It may be that there's existing IP for instance. There's an
FFT generator that comes with
Comes with the distribution
You can convert an FFT to convert a fourier transform into a male spectrum by combining bins
Maybe that's worth doing rather than writing a filter bank
But come talk
You've got plenty to invent without redoing the wheels
Any questions? I thought for a long time that I ought to
Tag final projects by the
By the techniques that they use but I never have I probably should start so ambient light from TV signal
Suggests that there's some sort of global
color analysis of the incoming signal
camera based
fingertip detection is a combination of video input and a bunch of
Morphological operations to look at the curvature of a fingertip and find out figure out where it is and use to control a video game
audio signal visualization
was an attempt to
convert
music to graphics
RGB LED matrix audio visualizer is really an exercise in
Is seeing how fast you can
Seeing how fast you can
How fast you can
run a matrix of six 64 by 64 matrix of LEDs and
Because you have so much brute IO on on on these boxes you can do quite reasonable animation
Even though you're outputting
lots of channels
Levitating a ball
was an effort to
Have a high-speed controller lifting a steel ball off the table and holding it in space using an electromagnet
So it's some power electronics. It's high-speed control loop
Some PID controllers and some graphics to debug the whole thing
Gesture Tetris, I don't even remember what that is. Oh
Oh
That's right
Steering it by moving your hands. So this was doing skin color detection
so this is detecting the color of the students hands and
She thoughtfully wore a purple sweatshirt that looks nothing like skin color
To help isolate her hands
cricket 2007 auto batter this was a
It was one of those projects which is so hard I was I was sure it wasn't gonna work except they got it working and
so they had to watch a VGA signal which meant they had to build a
custom board for for doing the high-speed ADC on three channels and
Hook that all built so they built a circuit board hooked it to the
Hooked it to the FPGA and came up with an algorithm watching the television signal to
to swing the cricket bat optimally to make lots of runs and
It was inhumanly good
Evolution game. I don't want to talk about
table tennis tracker
got
so they they figured out a way of setting up a
ping-pong table in the
MN lounge annoyed a lot of the image students
But but the but the FPGA could count could track the green ball
I don't know if we could see it on here or not. There it is. That's the
video and
It's automatically detecting the trajectory of the ball and what's happening with the ball
that made a whole bunch of websites including I triple-e computer as
a one pager
describing the project a
Rubik's Cube solver
With with little mechanical hands that rotated the Rubik's Cube. Ha ha ha video analysis
One of my favorites was a gravity and part and particle simulator because I like computational physics
this did a rock-paper-scissors
detected the shape of your hand and
I think didn't cheat
gesture control of an iPhone Mandelbrot color modulation
inventing the
personal computer
Yeah, they didn't get too far on that
So this one was hilarious fruit ninja with video tracking you could you could
So it's tracking a hand very reliably in
in the
with lots of faces and other stuff in the background and what you're trying to do is to slice fruit in half before it
flew off the screen and
We managed to spoof it finally by doing a sort of a quick change out from fist to face by
Running a fist up over somebody's face and then pulling their fist away and it started tracking their face
But but it's really quite good at rejecting noise and finding the centroid of the of the hand
Flappy bird player was awfully hard to get moving Bruce in a box. Oh, yes
So
Julie Wong made a a
device that would track a green square on the screen make an estimate of its of its size and
orientation and render stuff on to the
So there's me rendered on to the she's having a little too much fun with this
Interrupt a couple of hardware
Processor kind of things hardware Ethernet
Implementation nobody the right mind would do that. Now egg crusher game. That was truly strange
Upper body motion tracking. This is pretty cool. Tell me about your upper body tracker
Here is a down sampled version
Average version of all the pixels that a qualified skin pixels as you can see it's tracking my head and my hands
pretty clean
projection of the head tilt
That's kept computed based on the centroid of the head and the body just follows the head
And I could do things like and they're all updated simultaneously so you could do things
It could be like a future convention of the project if somebody could
So so it's parsed into
Hand and head and then re-synthesized as a 3d line
Element there NTSC battle game
Augmented. Oh, this was this is very cool. So what they did was they
Using edge detection and other techniques to convert that into a line image that you can then
Use as a maze
So what you can't see from this is that the
Is that the
That and if you walk into the lab now and look at the left end of the lab
You'll see that maze still on the board
Because what they did was they just took a video camera and pointed at the whiteboard
You drew whatever maze you wanted on the whiteboard and then their algorithm would fire balls into the new maze
you could modify the maze on the fly to make it easier or harder harder, right and
So it was kind of a mixed reality augmented reality
maze thing where the dynamics was controlled by the by the computer, but the but the maze is just hand-drawn and the
The really the hardest thing was getting the lines
And getting the noise free because they're all different
Intensities and different widths and ugly and they managed to make them all look like solid barriers
Let me see if I can show you
Very nice
Show the effect of changing the threshold right as you touch it, so we can't increase the threshold
Try to eliminate some of the specs. Uh-huh. The only consequence is you might lose
life I see so you might I
Want to try and find the picture? Okay. Oh, that's
That's the that's the one of the
That's a human with the same algorithm
Look at the eyeballs really stick
Okay, give me a different threshold now, all right, let's go up a little bit
So you can't really see all the fears, but you can still see like the outlines, right?
So it really depends on the application. Yeah, if you want to see details
Probably aren't gonna want to use it
So there's the camera. It's saying it's saying a video of the board over there. I
See so you you just freehand drew a
map on the board
And then you're using edge detection and other techniques to convert that into a line image that you can then
Use as a maze. Yeah
That's cool
Anti Tetris very odd
video game enhancer allowed you to
Do split screen on on?
Commodity video games
Hardware based AI for Super Mario Brothers
Turns out that's a hard thing to do
They managed to get it to some extent beat program will gate array
piano hero with virtual keyboard
Prime number generator and RSA encryptor decryptor. They were breaking passwords like crazy. It had like 40
RSA
Brute force
Searchers going. Oh, yes. I have to show you one more falling sand game. This is a cellular automaton
done by Skylar Schneider
Let me let me move up to it so every point on the on the
Pile
Similarly can start drawing here with the water
Now I can show you a bit about how these particles interact with each other
So for example, if I take the salt and the water here
They will mix together the water will mix with the salt to make it wet make salt water
Sand is insoluble in water. It will sink to the bottom
Similarly
By form a nice pool of oil here and try to drop some water into it
The oil rises to the top of the water
on the other hand
they draw
Well to connect the salt and the oil there's really no preference for one over the other they don't
Sink into one another they just kind of form their own piles
The same thing so this is doing a 640 by 480
Automaton with fairly complicated rules at at full at full
video rate
He spent a long time with pencil and paper
Doing the timing diagrams for that to show that he could get all of the cycles fit in that he needed for each pixel
So it's a very interesting staggered pipeline to do that
So I strongly suggest you take a look at some of these these
these examples because you'll give you some idea how to how to
hook stuff up real-time face tracking green screen chroma keying for
Being your own weather person
to the D graphic guitar synthesis
speech or a speaker recognition they managed to the two students managed to
Most of the time get the FPGA to tell who is speaking
The general algorithm for that by the way is to take the male spectrum
So you have 32 points coming out you take the discrete cosine transform of the male spectrum
If you take the spectrum of a spectrum and the the
the
lowest 12
coefficients of the
Cosine transform of the
male-transform
Are highly predictive of who is speaking?
It's called by the way. It's the spectrum is cepstrum. It's called the male cepstrum
it's the cosine transform of the male spectrum and
You can but by doing some by doing some training on a network
You can find out figure out who's speaking from those parameters which somehow represent the shape of your throat
the size and shape of your throat
So any questions? I mean, there's lots of stuff to look through
but again, the main thing about what I'm trying to get to here is talk to me or
Or look at the web page before you start reinventing algorithms so we could try and
Expedite some of the final project make it more efficient to do the final project development
Okay
So as of Wednesday
We are done here. We will be in lab
Starting this Wednesday starting this Wednesday. We'll be in lab. So thank you video note
and
Exactly zero of you have signed a release for the video from last week. Therefore none of the video will be up
Monday was the deadline for that. Is anybody actually going to do the release or no?
hmm
All right, no, okay, I guess we'll pass on that. All right. Thanks
