Capturing data from CAN-View

Wow! Thanks for the info, Roper! Exactly what i'm lookin for, and I'm sure others as well. As you mentioned, now we need a program to process the data.

 

<div class='quotetop'>QUOTE(roperld @ Mar 29 2006, 10:58 PM) [snapback]232144[/snapback]</div>

hello Dave thanks for the link.
I used seriallogger to collect data but realterm is free.
It is easy to read the log file from excel and to convert the data in hex or dec.
here is the code I wrote some time ago for interpreting the AVCLAN codes
in this code everybody can find the basic tricks to read a log file and put the interesting data in the right columns. after that, it is easy to put them on graphs.

 

<div class='quotetop'>QUOTE(roperld @ Mar 30 2006, 07:26 AM) [snapback]232268[/snapback]</div>

1) these are some data I collected for analysing the AVCLAN (Audio Video) to help norm understanding how the Nav Unit "speaks" to the MFD.
2) no object
3) I will come back later with an operational solution to show on the same graph MG2 rpm (speed) /ICE rpm / MG1 rpm & some other parameters. (no time before sunday!!!)
the idea is to collect the data during a trip and later on to show on graphs how they interact.

the macro I sent is be used by people with a good knowledge of excel/vba.

 

<div class='quotetop'>QUOTE(roperld @ Mar 29 2006, 02:06 PM) [snapback]231838[/snapback]</div>

I have been working with Dave for a few weeks on this and I now have a program that is available to convert the data.

My CAN-View decoder program reads the binary file that RealTerm created. It must contain RAW data (not formatted) from CAN-View. The program decodes 28 of the most interesting fields from the CAN packets and creates a .csv file with the data. You can then load the CSV file (here's a sample) into a spreadsheet program (such as Excel) and create graphs and do analysis and play with the data.

The following web page has a description of the program, download links, and descriptions of the fields that are output. The program is free and the source code is available for download as well.

CAN-View Decoder Program Web Page

Suggestions and comments are welcome. I'm not a windows programmer (I work on embedded systems) so I didn't do a Windows user interface, but the program works.

--GaryM

 

Dave, Gary, or anyone else that can help a "techno-peasant" with data capture:

I've tried Dave's instructions with the Realterm application and Gary's decoder for my new Can-view, and I'm having some problems. Reader's Digest version: After (presumably) logging data for 30+ minutes, the resulting CSV file has only 12 lines, with several blank columns and other columns with obviously invalid data (e.g., battery temp of 224C).

The long version:

No problems downloading and installing the Realterm application. I got the shortcut configured with the parameters as Dave describes. Realterm opened with no problem. My first attempt at capturing data generated an I/O error 103 (or something to that effect). A quick web search revealed that error is often related to not being to find or open a file, so I created the Prius folder on C: drive and ... voila! The error went away. So far, so good. Woohoo!

After giving Can-view the MFD command to start sending data, I started seeing what I assume are binary characters (i.e., symbols and gibberish) in the Realterm display window. I clicked the Start:Overwrite button and, interestingly, the characters stopped scrolling. But then the character counter in the lower left corner started. I figured that whether data was displayed or written to a file was an either/or thing. I rode for a half hour or more and then clicked Stop Capture. The character count was in the 350K range.

I later ran the Can Decoder application, with the expected prompt within the DOS window to enter the input file. I entered c:\prius\test and <enter>, whereupon multiple lines rapidly displayed and the window closed, all within a fraction of a second. Upon opening the resulting CSV file, the results were as described above. I re-ran the Can Decoder repeatedly to try to visually capture some of the text in that briefly appearing DOS window. Partial results (I think): "**** Warning, bad data, field out of range" followed by what seems to be a different field name on each line.

Any suggestions? Thanks in advance!!!

 

<div class='quotetop'>QUOTE(JimboK @ Jun 5 2007, 03:03 PM) [snapback]456025[/snapback]</div>

When you start recording it does stop displaying the characters, that's normal.
Though you should not see "gibberish" when not recording, you probably have the character set wrong.
Use the HEX (space) setting, there are multiple HEX settings, one of them should result in something that looks like this:

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 34 56 76 H4
A8 B3 C8 00 00 00 00 00 00 34 56 76 H4 A8 B3 C8 00 00 00 00 00
00 00 00 34 56 76 H4 A8 B3 C8 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 More numbers and letters from A-F 00 00 00 00 00
00 00 00 00 00 the 0-9,A-F is "HEX" a base-16 number system 00
00 00 etc etc scrolling down or up the screen...

This should start and stop scrolling as you tell the CAN-View to start and stop sending data....
Once you see this stuff you're ready to try recording and all should be well...
This is what the display=2 setting is supposed to do, perhaps it's not correct?

ps. Thank you Gary and Dave for the decoder app, and it's source! I'm thinking that it would be swell to setup a web script at the eaa-phev wiki site ( and elsewhere ) such that someone could create an "article" containing the raw hex and get it parsed and outputted to a table and multiple types of graphs, etc... As you can see from my graphs each one is different, so this is a project that I hope to get to work on sometime...

 

<div class='quotetop'>QUOTE(D0li0 @ Jun 8 2007, 08:39 PM) [snapback]458379[/snapback]</div>

Ryan, thank you very much for your detailed response. I haven't had time to do any more troubleshooting, though I did hear back from Gary Morris by e-mail. He was gracious enough to have me send him my raw binary file and the output file from Realterm so he could peek at them. His conclusion: "So you are not getting a good capture of data, it's either the data rate is mismatched or the capture program isn't working, or the cable, or the CAN-View is not in the right mode."

It probably boils down to a common problem I have with computers: a loose screw about two feet above the keyboard. I'll keep working on it.

 

OK, we're making headway. Woohoo!

Ryan, I had the Hex[space] setting selected before your post. Interestingly, what I've since discovered is that if I select one of the other "Display As" options, then go back to Hex[space], I now see the hex values as you describe. That alone didn't work, but at least the display behaved as you describe. I have to repeat that routine each time I open Realterm; else I see the characters and gibberish I first described.

Based on Gary's and Ryan's responses, I double-checked the settings and data rates; they were set properly. I've been using an old serial cable, so with the possibility that it could be bad I bought another. Meanwhile, I had some e-mail dialog with Norm to help verify the correct Realterm settings. His response: "You should be setting 8 data bits, parity, 1 stop bit (with parity as odd/even/space whatever, anything as long as it knows there is that 9th bit)." My settings matched all that except parity, which I had set to None.

So I changed that, plugged in the new cable, pushed all the buttons, and held my breath. The result: good data! Double woohoo!!!

Some lingering questions, if anyone has answers. Norm's site says, "The RAW message dump is a contiguous 58mS burst of (currently) 55 CAN packets each of 11 bytes, roughly once per second." From my brief driveway testing this evening, the resulting CSV files have somewhat less than 1 line per second, maybe 40-45 per minute. Does this seem about right? Can I assume each line represents one data burst from CAN? For later analysis of events that I'd like to put a precise time stamp on, is the time between each data burst exactly the same? The wording "roughly once per second" could be taken to mean it doesn't.

Thanks again to all! This is going to be fun once I start logging real world data!

As an aside, Norm tells me his father recently died. I know many of you think highly of him -- heck, I do without having met him -- so I thought you'd want to know.

 

<div class='quotetop'>QUOTE(JimboK @ Jun 15 2007, 03:36 PM) [snapback]462663[/snapback]</div>

I'm sorry to hear that. I know that his father was going to have some testing done several weeks ago, but hadn't heard any more. Thanks for letting us know.

Dave M.

 

Well I wanted to start capturing data with my Can-View and I thought it was working before but I guess I thought wrong.

After trying to capture data the program to turn it into a cvs only converts 99 lines. Also when doing a short capture of say about two minutes there are only about 10 lines or so of data.

The realterm program shows the hex before I start the overwrite but I noticed that the break and error lights are always on or the error will flash ever two or so seconds. Also when I tried to uncheck the direct capture box during capture it will display garbage instead of hex as well as give a warning type sound every second the packet of information is received from the Can-View. I have realterm setup just like the tutorial but cant seem to figure out what the problem is.

Does anyone have any ideas.

 

<div class='quotetop'>QUOTE(TheForce @ Oct 17 2007, 04:50 PM) [snapback]527071[/snapback]</div>

Jay, the symptoms that you describe with the data capture are not the first time publish here, it has been another thread referring to the same problem, for what I can remember, are the settings of the capture program, it been a few months back, do a deep search in PC.

 

Well I cant find it and I used all the search terms I know to use on both PC and Google. Thats strange because I'm pretty good at finding stuff. I guess it will be lost until I stumble on it or someone points it out to me.

Can you tell me if its normal for the break and error lights to be on? No matter what I try I can only get 1 line if I capture for 10 seconds. Shouldn't I be getting 1 line of data for every second?

 

<div class='quotetop'>QUOTE(TheForce @ Oct 17 2007, 07:27 PM) [snapback]527120[/snapback]</div>

The LEDs should be like dancing while at work.

This was the thread, read it from the beginning and you may find the answer ; dOliO had a good pointer. Have you replaced your serial cable? Hope this helps

 

I thought you might have pointed me to this thread but since I tried what was here I was still searching elsewhere.

Yup replaced the serial cable as well as trying the different parity settings. Maybe I have two bad serial cables? I'll have to see if I can find another one.

The thing is is that I can see the correct hex as it comes in but when I record I don't seem to get all the data. I could record for 30 seconds and then convert it to cvs and it will only have a few lines of information. Shouldn't it have one line of data for every second of recorded data? And if not why would it only convert 99 lines of data and quit?

 

Jay, I'm sorta glad you're having the problem. I am too. Makes me feel a little less inferior that someone with your technical smarts isn't able to get it working.

Ryan (aka, d0li0) tried (unsuccessfully) to help me troubleshoot it at Hybridfest, and more recently tekn0wledg (not a CV user, but a programmer familiar with communication software) has tried to help here locally. Tek has given me some additional troubleshooting tips that I haven't had the chance to try yet. (Frankly, I've been preoccupied with buying a house.) Please keep me posted if you're able to resolve it.

Horacio, was there supposed to be a link in your post? Was it, by chance, the thread where I first described my issues?

 

<div class='quotetop'>QUOTE(JimboK @ Oct 17 2007, 10:05 PM) [snapback]527170[/snapback]</div>

Yea, this was the thread in the back of my mind last night.

After trying to help out Jay on the hardware side on things that he already wisely checked, was typing some more small guidance and lost PC connection o the server hang-up again. I was ready to pinpoint him to the capture program (realterm?) settings or probably limitations or probably in need of an update, unfortunately I do not have it nor I used yet. But by all means I will keep an eye on the thread for some real progress and others headache relief.