More on reg exps. Parsing a C or C++ file with regexps in Python

Ok, so while i'm on the regexp wave. It's plain dumb but works fine. I know i could have used ctags or smth else, but for the sake of it. Also I googled for an example like this one but found nothing. It's always a good idea to have smth like this i case you need to generate prototypes for a long C file or nomatter what.

 #!/usr/bin/env python
 
 import os, re, sys
 
 class Cexploder():
     Body = re.compile("{[^{}]*}", re.DOTALL|re.MULTILINE)
     Func = re.compile("(\w+\s+)*\w+\s*\(.*?\)", re.DOTALL| re.MULTILINE)
     MultiLineComment = re.compile("/\*.*?\*/", re.DOTALL|re.MULTILINE)
     OneLineComment = re.compile("//.*?$", re.DOTALL|re.MULTILINE)
     CppDirectives = re.compile("^\s*#.*?$", re.DOTALL|re.MULTILINE)
     #^\s*#(.*?(\\\n)*)+\n
     
     def __init__(self, File=''):
         f = open(File, 'r')
         self.data = f.read()
         f.close()
 
     def GetPrototypes(self):
         FuncList = []
         pdata = self.data
         pdata = self.OneLineComment.sub('', pdata)
         pdata = self.MultiLineComment.sub('', pdata)
         pdata = self.CppDirectives.sub('', pdata)
 
         while self.Body.search(pdata) != None:
             pdata = self.Body.sub('', pdata)
         
         fiter = self.Func.finditer(pdata)
         for i in fiter:
             FuncList.append( pdata[i.start() : i.end()] )
         
         return FuncList
 
 c = Cexploder(sys.argv[1])
 f = c.GetPrototypes()
 for i in f:
     print i

Regular expressions for fun and profit!

So the year is 2007 the century 21-th, but still there are ppl who deny using regular expressions to do the job. Why? I don't know. May be they just don't know how to do it, may be they think it's slow, or they just never heard of it. But it makes me sick looking at lame pseudo parsers their awkward logic and tons of stupid useless code around for something as simple as digging a file for a pattern. I've seen guys spending a day writing code that can be replaced with just one line using a regular expression, and why? Do they gain something out of this? Performance? Speed? No, they usualy do this for a lame script to automate something and they don't care much if it's gonna take 10mS ot a second. The truth is they realy don't know how much easier it just because they have never done it before.

So for all of them here's a example of a quick and dirty script I used to generate a report. It simply goes through every file in a directory and looks for the following pattern "/*#.*?#*/". It's a bracket structure taken as a comment within C files. It's simple and i use it to write stuff about something that later might end up in a report. Sometimes i write in there something like "See line 1923" or "Line 192" wich ofcourse means a reference to that line, that's why i made my script look for such patterns and if one exists it'll dump it.

So enough goofing around this is the script:

 #!/usr/bin/env python
 
 import os, re, sys
 
 Cover = re.compile("/\*\#.*?\#\*/", re.DOTALL|re.MULTILINE)
 SeeAlso = re.compile("(see|line)+.*?\d+", re.DOTALL|re.MULTILINE|re.IGNORECASE)
 Number = re.compile("\d+")
 
 class Result:
     pass
 
 def GetLinesAround(Data = '', Line = 0, LinesAround = 5):
     ret = ''
     d = Data.split('\n')
     
     if Line >= LinesAround/2:
         upper = Line - LinesAround/2
     else:
         upper = 0;
     
     lower = Line + LinesAround/2 + 1
     if lower > d.__len__():
         lower = d.__len__()
     
     try:
         d = d[upper : lower]
     except Exception: 
         return ret
 
     for s in d:
         ret = ret + str(s) + '\n'
 
     return ret
 
 def GenUtReport(File = '', RelativePath=''):
     f = open(File, 'r')
     data = f.read()
     f.close()
 
     ret = []
     
     utIter = Cover.finditer(data)
     for i in utIter:
         res = Result()
         
         avLine = data[0:i.start()].count('\n') + (data[i.start():i.end()].count('\n')/2)
         
         res.fInfo = "File: " + str(RelativePath) + " Line: " + str(avLine)
         res.utInfo = data[i.start():i.end()].split(':')[1].replace('#*/', '')
         res.codeAroundUt = GetLinesAround(data, avLine)
         
         res.seeAlsoInfo = []
         seeIter = SeeAlso.finditer(res.utInfo)
         for si in seeIter:
             seeLine = Number.findall(res.utInfo[si.start():si.end()])
             if seeLine != None:
                 seeLine = seeLine[0]
                 res.seeAlsoInfo.append( ( seeLine, 10, GetLinesAround(data, int(seeLine), 10) ) )
         
         ret.append(res)
         
     return ret
 
 def PrityPrint(ret = []):
     tmpStr = ''
     
     for res in ret:
         tmpStr = tmpStr + "<tr>"
         tmpStr = tmpStr + "<td>" + res.fInfo + "</td>"
         tmpStr = tmpStr + "<td>" + res.utInfo  + "</td>"
         tmpStr = tmpStr + "<td>" + res.codeAroundUt + "</td>"
         tmpStr = tmpStr + "<td>"        
         for i in res.seeAlsoInfo:
             tmpStr = tmpStr + "See Also. Code around LINE: " + i[0] + " (+/-)"+ str(i[1]/2) +"\n\n"+ str(i[2])
         tmpStr = tmpStr + "</td>"
         tmpStr = tmpStr + "</tr>"
     
     return tmpStr.replace('\n', '<br>')
 
 tmpStr = """<html><head></head><body><table border="1">
 <tr>
 <td><b>Filename, Line number</b></td>
 <td><b>UT not covered reason</b></td>
 <td><b>Related code</b></td>
 <td><b>See also</b></td>
 </tr>"""
 
 for r, d, f in os.walk(sys.argv[1]):
     for file in f:
         tmpStr = tmpStr + PrityPrint(GenUtReport( os.path.join(r, file) , file))
 
 tmpStr = tmpStr + "</table></body></html>"
 print tmpStr

So you see in less than 100 lines I did it all. Parsing, referencing, generating a report in html. It took me less than 2 hours to finish this.

FuzzXml

Ok si i was playing around the other day with a Jabber server written in java. I don't quite remember the name but it had a nice web configuration interface. I remember thinking "why not just throw some garbage at this little server of ours?" Yeah sounds like a nice idea! Lets proceede in python of course. First we know jabber is based on XML, second we know it can work without encryption. How much easier could it be? I made a list of all the xml files i could find in my home dir. From all the browsing on the net for this year and a half since i got this laptop i had a hell of alot of them. I did "cat" them all into one big.blob, then refined it with "strings" and the resulting pile of XML stored as good.blob.

serj@tokamak:~$ du -h good.blob
2.4M good.blob

Now all i need is a bit of python to make this work.

#!/usr/bin/env python

import sys
import time
import random
import socket
import string
import threading

def fuzzFile(file, iter, chunkLen):
 f = open(file, 'r')
 data = f.read()
 f.close()

 retStr = ""
  for i in range(0, int(iter)):
      x = random.randint(0,data.__len__());
     retStr = retStr + str( data[  x : x+random.randint(0, int(chunkLen))  ] ) + "\n\n"
     for i in range( 0, int(random.randint(0, 3)) ):
         retStr = retStr + "\n"

 return retStr


class FuzzAServer(threading.Thread):
 def __init__(self, addr, port, file, iter, chunkLen):
     threading.Thread.__init__(self, name="Producer")
     self.addr = addr;
     self.port = port;
     self.file = file;
     self.iter = iter;
     self.chunkLen = chunkLen;

 def run(self):
     while 1:

         try:
             sox = socket.socket ( socket.AF_INET, socket.SOCK_STREAM )
             sox.connect((self.addr,int(self.port)))

             while sox.send(fuzzFile(self.file, self.iter, self.chunkLen)):
                 pass

             sox.close()
         except Exception, err:
             print err

         time.sleep(10)


class FuzzAClient(threading.Thread):
 def __init__(self, addr, port, file, iter, chunkLen):
     threading.Thread.__init__(self, name="Producer")
     self.addr = addr;
     self.port = int(port);
     self.file = file;
     self.iter = iter;
     self.chunkLen = chunkLen;

 def run(self):
     while 1:

         try:
             s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
             s.bind((self.addr, self.port))
             s.listen(1)
             conn, addr = s.accept()
             print 'Connected by', addr
             while 1:
                 data = conn.recv(1024)
                 print data,

                 if not data:
                     break


                 data = fuzzFile(self.file, self.iter, self.chunkLen)
                 print data

                 conn.send(data)

             conn.close()
         except Exception, err:
             print err

         s.shutdown(2)
         s.close()

for i in range(0, 1000):
 t = FuzzAServer(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5])
 t.start()
#    print "thread ", i

The idea is simple. Connect to the server build some random string of XML-ish garbage and send it back. And like this as fast as you can. And yeah lets spawn a couple of more in parallel.

As you can see i also included a server who can handle a client a time and send bad XML. I decided to test not just the server but the clients also.

The results? Well after about 10-30 seconds the java Jabber server eat about 1.2G ram and 98%cpu for more than 20 minutes (at least i killed it after 20 mins). The clients that i tested were examples from the XmlRpc++ library. The did handle the server without problems.