Open Source Automated Documentation in a Development Environment

Using Open Source Tools
for Document Automation

Neale Morison
May 2012

Who Am I?
• Various jobs
• Technical writer
• Journalist
• Editor
• Web developer
• Software developer
• Electronic technician

• Various employers
• Startups – hardware and software
• IBM, Cisco, Microsoft, Fujitsu…
• Today, CSIRO:
Commonwealth Scientific and Industrial
Research Organisation

CSIRO. Using Open Source Tools for Document Automation 2

What is ASKAP?

• Australian SKA
Precursor
• CSIRO radio telescope
36 antennas in Western
Australia
• World’s fastest survey
radio telescope
at 0.7-1.8 GHz
• The system will process
2Tb/s from each antenna
• Processed data then • What will it do?
shipped via fibre to a Perth • Discover millions of new galaxies
supercomputer - exascale
• Investigate dark matter, dark energy
• Similar issues to LHC
(Large Hadron Collider)

What is ASKAP?


Who Are You?
• Technical Writers
• Experience with:
• Web development
• Software development
• Hardware development
• System administration
• HTML
• XML
• JavaScript
• Python

What am I Talking About?
• Automation
Any kind of software or system that assists in
technical communication

• Open Source Tools
Broadly defined:
• Software: with an Open Source License
• Standards: non-proprietary or open
e.g. PDF, open since June 2008
• Languages: Python, JavaScript
(ECMAscript), HTML, XML,
• Apps: Apache HTTP server, MySQL

• Free Tools
Free (as in beer) apps – e.g. web browsers

Prologue - Fairlight 1985
• Fairlight CMI
(Computer Musical Instrument)
Series III
• Manual manual
literally cut and paste
• Index created automatically
• Simple algorithm

sample index.txt: sample output:
using code 32 a blog, writing 128
making an index 42 an index, making 42
writing a blog 128 blog, writing a 128
code, using 32
index, making an 42
making an index 42
using code 32
writing a blog 128

CSIRO ASKAP Antenna IDs – Problem

• Two different
identification systems
• Manufacturer: CETC54 ID
• ASKAP pad ID
• Tests refer to both

• Almost insoluble
• Which one to use?
• Why not both?
• Create a table showing the
correspondence


CSIRO ASKAP Antenna IDs

• Requirements
• Show ID correspondence
• Sort by either ID
• Link to test directories
• Show other useful information
• Portable – all contained in a
single small file
• Easy to update – antenna data
keeps changing


CSIRO ASKAP Antenna IDs - Solution

• Dynamic Document
• Fixed (static) HTML:
more maintenance
• Better to write JavaScript that writes the
HTML for you
• You have to maintain only the data:

tableRows = [
{'cetc54':1, 'askap':29, 'testDir':'ASKAP29_C01', 'FAT':'Y', 'SAT':'Y', 'MRO':'Y',},
…
]

• Responds to user requirements
• Smaller than static HTML
• ASKAP Antenna Table
• Filtered ASKAP Antenna Table


CSIRO ASKAP Antenna IDs - Solution

• Dynamic Document
• Fixed (static) HTML:
more maintenance
• Better to write JavaScript that writes the
HTML for you
• You have to maintain only the data:

tableRows = [
…
]

• Responds to user requirements
• Smaller than static HTML
• ASKAP Antenna Table
• Filtered ASKAP Antenna Table


Principles of Open Information
• Open
Store information in an open format
• Terse
Provide only the necessary information
• DRY
Don’t Repeat Yourself - Single Source [3]
• Accessible
Easy to find, easy to use
• Free
As in speech, love, beer – technology, IP
• Re-usable
• Maintainable
• Simple

Cisco Radiata Group

• WiFi on a chip

• Hardware startup exploiting:
• wireless invention
by John O’Sullivan and CSIRO radiophysics team
• analog IC design innovations

• Sold to Cisco in 1999 for unheard of sum
(around $400 million)


Cisco Radiata Register Docs – Problem

• Not really a problem – an opportunity

• Developers used a single source register description
in a Python script:
• generate XML register description
• generate Verilog code from which IC is built

• Required – a document generated from XML


Cisco Radiata Register Docs - Solution

• XML can be reformatted using XSLT and XSL/FO
(Extensible Stylesheet Language / Formatting Objects)

• Processed with Java tools, e.g. Saxon, Xalan

• Various outputs:
• HTML
• Postscript -> PDF
• LaTeX
• FrameMaker - MIF

• Result: a 100 page register reference manual
in seconds
• Register document example

Cisco Radiata Register Docs - Principles
• Open XML
• Terse Describes only the registers

• DRY One XML file stores all information

• Accessible one pdf

• Free common, available technology:
• Adobe Reader, Web browsers
• XML / XSLT
• HTML

• Re-usable use for next chip

• Maintainable
change only if XML format changes

• Simple for the users


Cisco Radiata Test Data – Problem

• Tests run on chip simulation

• Result: large quantity of numerical data

• Difficult to determine how data fits
specification

• Provided visualisation facilities do not
meet requirement


Cisco Radiata Test Data - Solution

• Process the data to
produce a customised
visualisation

• Tools used:
• Perl
• Lisp-like script
• Gnuplot

• Many tables, plots,
in HTML document
• Test data
visualisation example

Cisco Radiata Test Data - Principles
• Open Proprietary, but documented

• Terse No repeated data

• DRY Everything generated
from single set of data

• Accessible
HTML presentation

• Free
common, available technology:
• Web browsers
• Perl, HTML
• GnuPlot

• Re-usable use for next tests

• Maintainable
• Simple for the users

VaST XML Reports
• Software models of embedded systems, system-on-chip
(SoC)

• Used by developers of cell phones, automotive systems,
handheld devices to model the chips they were designing

• Low-level, cycle-accurate model that ran at speeds
comparable to the chip, sometimes faster

• Huge savings in time and money – chip developers could
write the firmware and software for the chip before
fabrication

• Integrated Development Environment (IDE) ran on
Windows, now also Linux.

• Acquired by Synopsys in 2010

VaST XML Reports – Problem

• Models shown as a hierarchy
obscures connections

• Item details hidden
multiple clicks, right clicks required

• No way to print full details
or to view as a whole

CSIRO. Using Open Source Tools for Document Automation

VaST XML Reports - Solution

• Data stored in XML

• XML can be:
• transformed using XSLT
• output as HTML in a browser

• With a single button
click, a full model report
can be instantly
generated:
PMX report example


VaST XML Reports – Good News,, Bad News
• Tricky
to get it to work - XLST

• Co-operation required
from GUI developers

• You only have to do it once

• Works when you’re not around,
for the customer’s models

• Maintenance: infrequent
when model storage format changes

• Value permanently added

VaST XML Reports - Principles
• Open XML

• Terse Describes only the model

• DRY One XML file stores all information

• Accessible one click

• Free common, available technology:
• Web browsers
• XML / XSLT
• HTML

• Re-usable
• Maintainable
• Simple for the users;
for the developer, not so much


CSIRO ASKAP Repositories – Problem

• SharePoint
• Corporate choice

• Subversion
• Developer choice

• Other repositories in use

• Don’t want to discourage or confuse

• Don’t want to introduce yet another system


CSIRO ASKAP Repositories - Solution
• Document Database
• Python CGI SQL app

• Scans repositories

• Can enter manually

• Captures metadata

• Single central site

• Searchable

• Links to issue tracker

• ASKAP Document
Database


CSIRO ASKAP Repositories - Principles
• Open SQL

• Terse normal form db

• DRY Single SQL db

• Accessible standard
web presentation, form fill

• Free
• Web browsers
• Python
• SQL

• Re-usable – other databases use same code
• Maintainable – some issues - complex to scan repositories. Code is documented.
• Simple – for users; but can I pass this on to another maintainer?


CSIRO ASKAP Document Management – Problem

• Want to track:
?
• Review History Tracking

• Approval ? ?
• History
Document
Repositories
Database
• Tie in to repositories
• Tie in to document database ?

• Complex and costly to program into database -
violates simplicity rule


CSIRO ASKAP Document Management – Solution

• Use issue tracking system
Redmine, Bugzilla… Redmine Issue &
History Tracking
• Perform review and approval in
URL ID
issue tracker

• Capture history in issue Repositories
Document
Database
• Link to repositories ID
via document URL

• Link from document database via document ID
• Link back to document database via document ID
• Requires procedures, training – document in wiki

CSIRO ADE Diagrams – Problem

• How Engineers
think:

• Requirements
• Hierarchy of
clickable diagrams

• Links from diagrams
to sub-diagrams

• Document for every block in diagram

• Links from diagrams to documents


CSIRO ADE Diagrams – Problem

• SVG
No – IE, XP

• HTML5
No – IE, XP

• PDF
Maybe, but
awkward to add links
clunky browser display

• Graphviz
Good range of outputs
Gets busy quickly: A B C
Not the style wanted
Links only in SVG output


CSIRO ADE Diagrams – Solution A

• Use hierarchy, not diagrams
• Procedure:
• Go through diagrams and
manually transfer nodes, links
to custom data format
• Use Python to read data and output JSON
• Use JavaScript to generate a hierarchy map from JSON
• Add document names, responsibilities, metadata
• Generate documents using PowerShell, COM objects 
• ADE Hierarchy

CSIRO ADE Diagrams – Solution B

• Do the diagrams
HTML Image Map, JavaScript

• Procedure:
• Create diagrams as simple graphics – PPT
• Save as individual images
• Create image maps – free program
• JavaScript code transforms simple image
map
• JavaScript links to the hierarchy page
• ADE System Block Diagrams

CSIRO ADE Diagrams - Principles
• Open Python, HTML, JavaScript
Doc creation not open, but free: PowerShell

• Terse Only data

• DRY One source

• Accessible standard web presentation

• Free
• Web browsers
• Python
• JavaScript
• PowerShell

• Re-usable
• Maintainable
• Simple

Conclusion
• Coding is there to be learned
• HTML, XML, JavaScript, C, Java, Perl, Python, SQL, LaTeX …
• Excellent online resources

• You have to learn something
– either a proprietary system, or open tools

• Advantages of open tools
• Skill Transfer
• Embody general principles
• Expert and helpful community

• You can solve otherwise intractable problems
• Free
The only cost is time


Glossary
• ASKAP Australian SKA Pathfinder
• CGI Common Gateway Interface
• DRY Don’t Repeat Yourself
• DRY Don’t Repeat Yourself
• FAT Factory Acceptance Test
• GNU GNU is Not Unix
• HTML HyperText Markup Language
• ID Identifier
• Recursion see Recursion
• SKA Square Kilometre Array
• SAT Site Acceptance Test
• SQL Structured Query Language
• SVG Scaleable Vector Graphics
• URL Uniform Resource Locator
• XML eXtensible Markup Language
• XSLT eXtensible Stylesheet Language Transformations
• XSL/FO eXtensible Stylesheet Language /Formatting Objects


References
• 1. Open Source Initiative
http://www.opensource.org
• 2. Fairlight Instruments Web site
fairlightinstruments.com.au
• 3. DRY – Don’t Repeat Yourself
The Pragmatic Programmer: From Journeyman to Master
by Andrew Hunt and David Thomas
http://pragprog.com/the-pragmatic-programmer
• 4. ASKAP
• http://www.atnf.csiro.au/projects/askap/
• 5. CSIRO
http://www.csiro.au/
• 6. CSIRO Wireless LAN patent


CSIRO Astronomy and Space Science
Neale Morison
ASKAP Documentation Manager

Phone: 02 9372 4726
Email: neale.morison@csiro.au
Web: www.csiro.au/org/cass

Thank you
Contact Us
Phone: 1300 363 400 or +61 3 9545 2176
Email: enquiries@csiro.au Web: www.csiro.au

Open Source Automated Documentation in a Development Environment

Recommended

More Related Content

Viewers also liked (12)

Similar to Open Source Automated Documentation in a Development Environment (20)

Recently uploaded (20)

Open Source Automated Documentation in a Development Environment