OT?: break up an incoming data stream into fixed length lines? in bash?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

OT?: break up an incoming data stream into fixed length lines? in bash?

Dave Stevens
I'm using U16.04 to read data from a sensor. The incoming serial data
stream has no newlines, just continuous bytes. In the datastream the
letters B and M occur together at the start of every 24 byte
subsequence. I'm reading the data using getserial with appropriate
speed and parity parms. I don't see a simple way to break it up into
lines with each line having 24 bytes. Does anyone care to suggest a
method? I'd use a bash function if there is one.

TIA,

Dave

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: OT?: break up an incoming data stream into fixed length lines? in bash?

Robert Heller
At Fri, 6 Apr 2018 11:04:08 -0700 "Ubuntu user technical support,  not for general discussions" <[hidden email]> wrote:

>
> I'm using U16.04 to read data from a sensor. The incoming serial data
> stream has no newlines, just continuous bytes. In the datastream the
> letters B and M occur together at the start of every 24 byte
> subsequence. I'm reading the data using getserial with appropriate
> speed and parity parms. I don't see a simple way to break it up into
> lines with each line having 24 bytes. Does anyone care to suggest a
> method? I'd use a bash function if there is one.

You are going to have to use low-level serial I/O.  I am presuming we are
talking about RS232/485 (/dev/ttySnn) OR a USB/Serial device (/dev/ttyUSBn or
/dev/ttyACMn).

Commonly, that would mean have way to much fun coding in C/C++, but it is
actually possible to do that with Tcl -- something I have done for various
serial-based Model Railroad interfaces.  (It might be possible with Python, but
I don't use Python, so I have no clue).

There is Tcl code libraries under

https://github.com/RobertPHeller/ModelRRSystem

That do low-level serial I/O in Tcl (eg read one character at a time).

Look in these sub-directories:

trunk/Scripts/CMri
trunk/Scripts/CTIAcela
trunk/Scripts/XPressNet

They all use much the same low-level I/O structure.

>
> TIA,
>
> Dave
>

--
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
[hidden email]       -- Webhosting Services
                                               

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: OT?: break up an incoming data stream into fixed length lines? in bash?

Thomas Kaiser (ubuntu)
In reply to this post by Dave Stevens
On 06.04.2018 20:04, Dave Stevens wrote:

> I'm using U16.04 to read data from a sensor. The incoming serial data
> stream has no newlines, just continuous bytes. In the datastream the
> letters B and M occur together at the start of every 24 byte
> subsequence. I'm reading the data using getserial with appropriate
> speed and parity parms. I don't see a simple way to break it up into
> lines with each line having 24 bytes. Does anyone care to suggest a
> method? I'd use a bash function if there is one.
>
> TIA,
>
> Dave
>

Hello Dave

I use following bash script to read data from a serial port (USB <-> RS232):

#!/bin/bash
# Receive remote weather data from USB-WDE1

# Set the correct interface parameters
stty < /dev/ttyUSB0 9600 -brkint -opost -onlcr -echo

# Loop forever to read data from USB-WDE1
socat /dev/ttyUSB0,b9600 STDOUT | \
while read line
do
      if [[ "${line%%;*}" == '$1' ]] ; then
        #echo $line
          # format data
          tmp=`echo "${line#?1;1;}" | tr ';,' ':.'`
          data=`echo "N${tmp%%0}" | sed 's/::/:U:/g' | sed 's/::/:U:/g'`
          data=${data%%:}
        echo `date +%R.%S:` $line
      fi
done

It is a long time since I implemented this with the help of examples from various web sites. Therefore, I don't remember how the data formating is done in detail.
Maybe this example gets you running.

Thomas

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: OT?: break up an incoming data stream into fixed length lines? in bash?

Robert Heller
At Sun, 8 Apr 2018 15:14:56 +0200 "Ubuntu user technical support,  not for general discussions" <[hidden email]> wrote:

>
> On 06.04.2018 20:04, Dave Stevens wrote:
> > I'm using U16.04 to read data from a sensor. The incoming serial data
> > stream has no newlines, just continuous bytes. In the datastream the
> > letters B and M occur together at the start of every 24 byte
> > subsequence. I'm reading the data using getserial with appropriate
> > speed and parity parms. I don't see a simple way to break it up into
> > lines with each line having 24 bytes. Does anyone care to suggest a
> > method? I'd use a bash function if there is one.
> >
> > TIA,
> >
> > Dave
> >
>
> Hello Dave
>
> I use following bash script to read data from a serial port (USB <-> RS232):
>
> #!/bin/bash
> # Receive remote weather data from USB-WDE1
>
> # Set the correct interface parameters
> stty < /dev/ttyUSB0 9600 -brkint -opost -onlcr -echo
>
> # Loop forever to read data from USB-WDE1
> socat /dev/ttyUSB0,b9600 STDOUT | \
> while read line
> do
>       if [[ "${line%%;*}" == '$1' ]] ; then
> #echo $line
>           # format data
>           tmp=`echo "${line#?1;1;}" | tr ';,' ':.'`
>           data=`echo "N${tmp%%0}" | sed 's/::/:U:/g' | sed 's/::/:U:/g'`
>           data=${data%%:}
> echo `date +%R.%S:` $line
>       fi
> done
>
> It is a long time since I implemented this with the help of examples from various web sites. Therefore, I don't remember how the data formating is done in detail.
> Maybe this example gets you running.

This code clearly assumes data coming in line-based records (eg CRLF between
"records" -- "while read line").  If the data stream is NOT line-based, this
code won't work.  *Sometimes* sensor instruments send data with newlines,
sometimes not... When not, things can become interesting to code for...  Even
more fun when the data steam uses non-printable ASCII characters or data
bytes.

>
> Thomas
>

--
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
[hidden email]       -- Webhosting Services
                                                                                               

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: OT?: break up an incoming data stream into fixed length lines? in bash?

Peter Silva
This is pretty straightforward stuff. What is needed is a complete specification
of the protocol that is used over the serial line.  Then you need to implement
the protocol.  This sounds like someone's homework assignment.

stuff I've written to do this sort of thing...

Here's some code that implements a GUI to talk to a Z-80 over a serial link.
It's in python2:

https://github.com/petersilva/Bi-Directional-Machining-Facility

Python is likely a very good approach, but you can use any language you want.

This other tool is in debian, drobo-utils.sf.net.  This is another
python app that is
communicating with an ARM processor over SCSI protocol inside a USB
link. In this case, the usb is transparent because other linux drivers
make it present a SCSI application driver (work with /dev/sdX).  In
any event, this stuff is fairly straightforward if you know the
protocol you need to use.

If you don't have a protocol specification, then it is orders of
magnitude more work, because you end up reverse engineering, and the
debugging sucks.




On Sun, Apr 8, 2018 at 9:38 AM, Robert Heller <[hidden email]> wrote:

> At Sun, 8 Apr 2018 15:14:56 +0200 "Ubuntu user technical support,  not for general discussions" <[hidden email]> wrote:
>
>>
>> On 06.04.2018 20:04, Dave Stevens wrote:
>> > I'm using U16.04 to read data from a sensor. The incoming serial data
>> > stream has no newlines, just continuous bytes. In the datastream the
>> > letters B and M occur together at the start of every 24 byte
>> > subsequence. I'm reading the data using getserial with appropriate
>> > speed and parity parms. I don't see a simple way to break it up into
>> > lines with each line having 24 bytes. Does anyone care to suggest a
>> > method? I'd use a bash function if there is one.
>> >
>> > TIA,
>> >
>> > Dave
>> >
>>
>> Hello Dave
>>
>> I use following bash script to read data from a serial port (USB <-> RS232):
>>
>> #!/bin/bash
>> # Receive remote weather data from USB-WDE1
>>
>> # Set the correct interface parameters
>> stty < /dev/ttyUSB0 9600 -brkint -opost -onlcr -echo
>>
>> # Loop forever to read data from USB-WDE1
>> socat /dev/ttyUSB0,b9600 STDOUT | \
>> while read line
>> do
>>       if [[ "${line%%;*}" == '$1' ]] ; then
>>       #echo $line
>>           # format data
>>           tmp=`echo "${line#?1;1;}" | tr ';,' ':.'`
>>           data=`echo "N${tmp%%0}" | sed 's/::/:U:/g' | sed 's/::/:U:/g'`
>>           data=${data%%:}
>>       echo `date +%R.%S:` $line
>>       fi
>> done
>>
>> It is a long time since I implemented this with the help of examples from various web sites. Therefore, I don't remember how the data formating is done in detail.
>> Maybe this example gets you running.
>
> This code clearly assumes data coming in line-based records (eg CRLF between
> "records" -- "while read line").  If the data stream is NOT line-based, this
> code won't work.  *Sometimes* sensor instruments send data with newlines,
> sometimes not... When not, things can become interesting to code for...  Even
> more fun when the data steam uses non-printable ASCII characters or data
> bytes.
>
>>
>> Thomas
>>
>
> --
> Robert Heller             -- 978-544-6933
> Deepwoods Software        -- Custom Software Services
> http://www.deepsoft.com/  -- Linux Administration Services
> [hidden email]       -- Webhosting Services
>
>
> --
> ubuntu-users mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: OT?: break up an incoming data stream into fixed length lines? in bash?

Karl Auer
In reply to this post by Dave Stevens
On Fri, 2018-04-06 at 11:04 -0700, Dave Stevens wrote:
> I'm using U16.04 to read data from a sensor. The incoming serial data
> stream has no newlines, just continuous bytes. In the datastream the
> letters B and M occur together at the start of every 24 byte
> subsequence. I'm reading the data using getserial with appropriate
> speed and parity parms. I don't see a simple way to break it up into
> lines with each line having 24 bytes. Does anyone care to suggest a
> method? I'd use a bash function if there is one.

Sounds like you need a little state machine.

You can do it in bash. Read the doco for the "read" function; it has
options for setting the number of characters to read. By default it
reads a line; to read a single char you may have to change some tty
settings - a bit of googling will find you what you need.

Your state machine would look something like:

   clear buffer
   state = waiting_for_B
   while not EOF
      get char
      if state is:
         waiting_for_B 
            if char == B
               append char to buffer
               state = waiting_for_M
         waiting_for_M
            if char == M
               append char to buffer
               state = collecting
            else
               clear buffer
               state = waiting_for_B
         collecting
            save char in buffer
            if buffer length > 23
               output buffer, newline
               clear buffer
               state = waiting_for_B
      
The above discards any data not inside a 24-byte sequence starting with
"BM". You can use the same state machine on data from a file.

Regards, K.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer ([hidden email])
http://www.biplane.com.au/kauer
http://twitter.com/kauer389

GPG fingerprint: A0CD 28F0 10BE FC21 C57C 67C1 19A6 83A4 9B0B 1D75
Old fingerprint: A52E F6B9 708B 51C4 85E6 1634 0571 ADF9 3C1C 6A3A



--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users