I have struggled with some similar problems already I believe discussed but really I do not get the way out: What if your file has data for example as below (hex values can be anything but fixed to 1 byte) and you want to convert each hex to decimal and keep it as in below format so instead of hex you display decimal and still comma separated? - Please note number of data lines is not fixed it can by any number.
Example 1:
0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a 0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a 0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a 0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 Example 2: same as above but lets say it has data name in front:
X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 So I have spent some time to understand the bash grep, sed etc.. as I am completely not Software guy I manage to play and filter out some text files in the correct format next I would like to understand how to operate on strings with more than one value and above examples will really help . Thanks in advance.
I did try printf, echo and I am able to use them when each string in text file has one value for example : if I use data file which has data
0xff 0xff 0xff 0xff 0xff . . . . . . .. 0xff all works fine and can be converted with printf function: this was found in other thread:
while read line do printf '%d' $line done < file Above works great for the one hex value in line: Can this be adjusted to work with my examples or there is any better way?
16 Answers
You can use (GNU) Awk and it's strtonum function, which converts strings that start with 0x to decimal numbers.
- Just splitting at
,and starting from column1:
awk 'BEGIN {FS=OFS=","} {for (i=1;i<=NF;i++) $i=strtonum($i); print}' file.txt 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 - Splitting at
,or:and starting from column2:
awk 'BEGIN {FS=",|:"; OFS=","} {for (i=2;i<=NF;i++) $i=strtonum($i); print}' file.txt X,81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y,90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z,240,240,240,240,240,240,240,240,240,240,240,240,240,240 X,81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y,90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z,240,240,240,240,240,240,240,240,240,240,240,240,240,240 X,81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y,90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z,240,240,240,240,240,240,240,240,240,240,240,240,240,240 - This way, however, also the
:will be converted to,. If this is not desired, you could print the columns1,2, and the rest separately:
awk 'BEGIN {FS=",|:"} {printf "%s:%s", $1, strtonum($2); for (i=3;i<=NF;i++) printf ",%s", strtonum($i); printf "\n"}' file.txt X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 A generalized awk solution -
awk -F'[:,]' -v OFS=, '{for(i=1;i<=NF;i++){if($i~/^0x/){$i=strtonum($i)}}if($1!~/^[0-9]/)sub(",",":");print}' file Saved to a file, run with each of your example sets:
$: cat x2d #! /usr/bin/awk -f BEGIN{ FS="[:,]"; OFS="," } { for ( i=1; i<=NF; i++ ) { if ( $i ~ /^0x/ ) { $i=strtonum($i) } } if ( $1 !~ /^[0-9]/ ) sub(",",":"); print; } $: ./x2d withName X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 $: ./x2d noName 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 Just for the practice, the same in basic bash.
$: cat x2d #! /bin/bash while IFS=:, read -ra field do for f in "${field[@]}" do case $f in 0x*) printf %d, $f;; *) printf %s: $f;; esac done printf $'\n' done < $1 $: ./x2d withName X:81,81,81,81,81,81,81,81,81,81,81,81,81,81, Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90, Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240, X:81,81,81,81,81,81,81,81,81,81,81,81,81,81, Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90, Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240, X:81,81,81,81,81,81,81,81,81,81,81,81,81,81, Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90, Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240, $: ./x2d noName 81,81,81,81,81,81,81,81,81,81,81,81,81,81, 90,90,90,90,90,90,90,90,90,90,90,90,90,90, 240,240,240,240,240,240,240,240,240,240,240,240,240,240, 81,81,81,81,81,81,81,81,81,81,81,81,81,81, 90,90,90,90,90,90,90,90,90,90,90,90,90,90, 240,240,240,240,240,240,240,240,240,240,240,240,240,240, 81,81,81,81,81,81,81,81,81,81,81,81,81,81, 90,90,90,90,90,90,90,90,90,90,90,90,90,90, 240,240,240,240,240,240,240,240,240,240,240,240,240,240, For data sets this small, the difference is irrelevant.
If the inputs are big, the awk will be faster.
I'd use perl. For your first case:
$ perl -F, -lane 'print join(",", map(hex, @F))' ex1.txt 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 and the second, with the prefix:
$ perl -F'[:,]' -lane 'print $F[0], ":", join(",", map(hex, @F[1..$#F]))' ex2.txt X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 A straightforward Bash solution would be to just let Bash parse integers like it normally does. For Example 2:
to_dec() { local -r IFS=,: local -a line local -ai nline while read -ra line; do nline=("${line[@]:1}") printf '%s:%s\n' "${line[0]}" "${nline[*]}" done } Then to_dec < file gives the expected output for Example 2. A to_dec for Example 1 will be much simpler, because there’s no need for an auxiliary array in that case; just read the numbers:
to_dec() { local -r IFS=, local -ai nline while read -ra nline; do printf '%s\n' "${nline[*]}" done } As for the math operations for Example 2, requested in a comment, in pure Bash one could create functions named after the line prefixes and then execute them on the arrays. In any case, it won’t be “efficient”; it’s Bash. (Also, presumably, Bash has integer division only; to circumvent that one could use a fixed-point representation of some sort, but that’s way too hacky in Bash.)
#!/bin/bash X() { local -n a="$1"; local -i i for i in "${!a[@]}"; do ((a[i] /= 2)); done; } Y() { local -n a="$1"; local -i i for i in "${!a[@]}"; do ((a[i] *= 2)); done; } Z() { local -n a="$1"; local -i i for i in "${!a[@]}"; do ((a[i] += 2)); done; } to_dec() { local -r IFS=,: local -a line local -ai nline while read -ra line; do nline=("${line[@]:1}") "${line[0]}" 'nline' # transform the array printf '%s:%s\n' "${line[0]}" "${nline[*]}" done } to_dec # takes the script's default stdin / stdout For a bit more efficiency and floating-point arithmetics, awk can help:
#!/usr/bin/awk -f BEGIN { FS = "[:,]" } function X(numbers) { for (i in numbers) numbers[i] /= 2 } function Y(numbers) { for (i in numbers) numbers[i] *= 2 } function Z(numbers) { for (i in numbers) numbers[i] += 2 } { for (i = 2; i <= NF; ++i) numbers[i] = strtonum($i) f = $1 @f(numbers) output = $1 ":" numbers[2] for (i = 3; i <= NF; ++i) output = output "," numbers[i] print output } For large data processing, a proper implementation in a more low-level programming language would be more efficient.
5GNU AWK
awk -F'[,:]' -v OFS=, ' /:/{ withName=1 } { for(i=1;i<=NF;i++) $i=(strtonum($i) ? strtonum($i) : $i) } withName{ sub(/,/,":") withName=0 }1 ' file1 file2 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 echo '0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a 0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a 0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a 0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0' | gawk -nb '$_ += ORS = RT' RS='\n|,' 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 81,81,81,81,81,81,81,81,81,81,81,81,81,81 90,90,90,90,90,90,90,90,90,90,90,90,90,90 240,240,240,240,240,240,240,240,240,240,240,240,240,240 gawk -nMb '$++NF = +$_' OFS='\f' 0x5151515151515151515151515151 1649317825652239364356745892811089 0x5a5a5a5a5a5a5a5a5a5a5a5a5a5a 1832575361835821515951939880901210 0xf0f0f0f0f0f0f0f0f0f0f0f0f0f0 4886867631562190709205173015736560 0x5151515151515151515151515151 1649317825652239364356745892811089 0x5a5a5a5a5a5a5a5a5a5a5a5a5a5a 1832575361835821515951939880901210 0xf0f0f0f0f0f0f0f0f0f0f0f0f0f0 4886867631562190709205173015736560 0x5151515151515151515151515151 1649317825652239364356745892811089 0x5a5a5a5a5a5a5a5a5a5a5a5a5a5a 1832575361835821515951939880901210 0xf0f0f0f0f0f0f0f0f0f0f0f0f0f0 4886867631562190709205173015736560 echo ' X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0 X:0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51,0x51 Y:0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a,0x5a Z:0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0,0xf0' | gawk -nb 'NF == +(ORS = RT) || $_ += _' RS='\n?[A-Z]:|[\n,]' X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 X:81,81,81,81,81,81,81,81,81,81,81,81,81,81 Y:90,90,90,90,90,90,90,90,90,90,90,90,90,90 Z:240,240,240,240,240,240,240,240,240,240,240,240,240,240 
