Ohler Dev Notes

peter@ohler.com

I'm Peter Ohler and this is my site. Have a look around and check out the open source projects linked by the large Os on the right side of the page. You can view my old pages by clicking on the Old link. If you want to contact me you can email me at peter@ohler.com.

opo

January 9, 2017


How about a graph database? Performance on the currently available graph databases or triple stores is not that impressive. It seems like an area worth exploring. opo is my attempt at triple store exploration. It's looking pretty fast so far. On the long term road map are support for RDF, TURTLE, and SPARQL.

The project re-uses some of the HTTP server code from Piper Push Cache and gets similar latency and throughput. Even with REST over HTTP and not a custom interface more than 100,000 retrievals per second are possible.

read more...

Piper Push Cache

December 15, 2016


Most of the systems I develop use push to move data around. JSON is another area I've accumulated a fair amount of expertise in. Putting the two together in a web server resulted in Piper Push Cache, a very fast scalable web server that has build in support for push in a number of ways. The performance surprised me as it performed better than nginx. Check out the benchmarks.

The first push technology is getting data in the form of JSON out to web pages. This uses my own WebSockets implementation. SSE is used for mobile browsers. The second use of push is getting data from the server to other applications. For this NATS is used.

The final bit of fun in Piper Push Cache is a process flow engine. The fun part of that was using Javascript to write a complete process flow editor.

read more...

GemChart, an OFlow Example

December 22, 2014


OFlow is the latest in a series of process flow tools I've worked on over the years going back to my first one at TIBCO Software. OFlow relies on queues, immutable data, and a separate thread for each task in a process flow. It is build to allow highly parallel processing.

GemChart is a somewhat non trivial example of how OFlow can be used. It also gave me an excuse to put together a graphing application for charting downloads of my gems. It was fun to put together and gave me a chance to test out the debugging and tracing tools in OFlow.

read more...

oDisk (thats Other Disk, not iDisk)

July 30, 2012


The disappearance of Apple's iDisk left a hole in my backup strategy that needed to be filled. I no longer had a place to backup my files and pull them onto multiple computers. iCloud, the iDisk replacement supports only Apple compliant product files and not arbitrary files. It was the perfect opportunity or rather and excuse to try out OPEE and see how well the alternative threading approach worked. As long as I was rolling my own solution for remote storage it might as well be encrypted and compressed. The result is oDisk.

read more...

Not Your Usual Threading

May 10, 2012


The Object-based Parallel Evaluation Environment or OPEE project explores an alternative approach to using multiple threads for system design. OPEE was purpose built to solve problem encountered in system that involve significant processing and shared resources. Shared resources are components such as file writers, hash tables, and summary data. A trading system design is explored in this article.

read more...

Need for Speed

March 13, 2012


After writing Ox, a fast XML parser, a fast JSON parser seems like the good next project to tackle. The goal was to come up with a parser that was as fast as possible while still providing a consistent and understandable API. The JSON parser, Oj was so much faster than any of the other JSON parsers that it seemed worth describing the process involved in how those results were achieved.

read more...

3+ Ways to Parse XML in Ruby

September 27, 2011


There are several ways to parse an XML document. Which one to choose depends on the application. Three approaches are explored here. All three approaches parse an Apple plist XML document and convert it into a Hash tree with native values and Arrays.

The first approach makes use of an in memory parser and then converts the resulting document Ruby Objects into a Hash. The second uses a SAX like callback parser to create a Hash tree directly. The third converts the XML to a serialized Object format and then loads that XML directly into a Hash. Performance and simplicity is considered in all cases.

read more...

Parse Approach Comparison

XML with Ruby

September 21, 2011


XML is a well documented, widely used, and well supported format for encoding data. It is heavily used on the web and with languages such as Java. XML format is stable and well documented. The downside is it is rather verbose but that is also what makes it readable by humans as well as machines.The wide support in almost every language for XML makes it extremely portable.

Until recently using XML with Ruby was not a high performance option. The two most common and highest performing Ruby Gems were Nokogiri and LibXML. Both provide parsing to Ruby Object representation of an XML document as well as a SAX like stream parser. The Ox gem was created to address the need for a more optimized XML parser so that the advantages of XML could be made available in Ruby without suffering the performance impact.

read more...

SAX Comparison

Parse Compare

Ruby Object XML Serialization

September 20, 2011


In an attempt to use a more portable Object serialization method than Marshal that performed as well or better, Ox was born. A Ruby implementation on top of Nokogiri was prohibitively slower than using the native Ruby Marshal module.

Marshal has a number of problem though. The format changes from one Ruby version to the next and it is not portable between any other languages. It is a binary format that is difficult to read, debug, or edit.

As a extension, Ox is able to dump and load a Ruby Object to and from XML faster than the native Ruby Marshal can. Ox also supports serialization of Exceptions better than Marshal.

read more...

Load Results

Dump Results

Oj

Fast JSON Parser

Ox

Fast XML Parser

OjC

Optimized JSON in C

Piper

Piper Push Cache
Why Wait, Push It

ODisk

Opee based
File Synchronization Application

OFlow

Operations Workflow
Highly parallel workflow

Old

Former Site
(Electric Cars, Robotics, Bicycles, and more)

Archive...