Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update readme file (corrected PR) #3

Open
wants to merge 3 commits into
base: release_0.5
Choose a base branch
from
Open

Update readme file (corrected PR) #3

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6476c21
Updated README. Converted to Markdown, added information about running
ms705 Jun 13, 2015
c6c40b1
Move README to .md file.
ms705 Jun 13, 2015
28583aa
Make README links into links in Markdown.
ms705 Jun 13, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 59 additions & 39 deletions README.txt → README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,31 +1,42 @@
Installation steps
------------------
# Overview
Naiad is a distributed, parallel dataflow computation framework. It supports
both batch and streaming computations, and is described in detail in the
[SOSP 2013 paper](http://dl.acm.org/citation.cfm?id=2522738).

# Installation steps

## Compatibility
Naiad works on Windows using the Microsoft .NET framework (v4.5+) and on
Linux using Mono 3.0+. Other configurations (e.g. the Linux version of the .NET
framework) *may* work, but are untested.

## Components
Naiad is distributed as a Visual Studio 2013 solution, comprising
the following projects:

* "Naiad", which contains the core Naiad system and support libraries,
and builds Microsoft.Research.Naiad.dll. If you create your own project that
* *Naiad*, which contains the core Naiad system and support libraries,
and builds `Microsoft.Research.Naiad.dll`. If you create your own project that
uses Naiad, you must add a reference to this library.

* "Lindi", a LINQ-like library for data-parallel iterative computation.
* *Lindi*, a LINQ-like library for data-parallel iterative computation.

* "DifferentialDataflow", a LINQ-like library supporting incremental
* *DifferentialDataflow*, a LINQ-like library supporting incremental
execution of iterative computation.

* "AzureSupport", a library of Naiad components that supports writing to
and reading from Windows Azure Storage.
* *AzureSupport*, a library of Naiad components that supports writing to
and reading from Windows Azure Storage (Windows only).

* "Examples", which contains several example Naiad applications and
builds Examples.exe.
* *Examples*, which contains several example Naiad applications and
builds `Examples.exe`.

* In addition, the "ClusterSubmission" solution contains experimental support
* In addition, the *ClusterSubmission* solution contains experimental support
for launching Naiad programs on Azure and YARN clusters.

These projects can be built and executed using any of the following
implementations of the .NET framework:

* Microsoft .NET Framework 4.5
* Mono v3.0+

To build Naiad, follow the steps below that correspond to your setup.

Expand All @@ -50,43 +61,50 @@ To build Naiad, follow the steps below that correspond to your setup.

3. At the command prompt, run MSBuild.exe.

Getting started with the example programs
-----------------------------------------
* Build using Mono (on Linux, without Visual Studio)

1. Open a terminal and navigate to the root directory of the Naiad
repository.

2. Run `./build_mono.sh`.


# Getting started with the example programs

As part of this source distribution, the Examples project demonstrates
how to write simple applications using Naiad. After building the Naiad
solution, you will find the executable Examples.exe in the
Examples\bin\Debug directory.
solution, you will find the executable `Examples.exe` in the
`Examples\bin\Debug directory`.

Examples.exe uses command-line arguments to select an example, and it
`Examples.exe` uses command-line arguments to select an example, and it
is recommended that you run it from a command prompt or terminal. For
example, to run the word count example, follow these steps:

1. Open a command prompt or terminal, and change to the
Examples\bin\Debug directory.
`Examples\bin\Debug` directory.

2. At the command prompt, type "Examples.exe" to see the usage and
2. At the command prompt, type `Examples.exe` to see the usage and
available examples.

3. At the command prompt, type "Examples.exe wordcount" to get started
3. At the command prompt, type `Examples.exe wordcount` to get started
with the wordcount example.

Known issues with Naiad
-----------------------

# Known issues

This release of Naiad is an alpha release. The amount of testing the code
has been subjected to is limited, mostly to programs we have written. The
email address [email protected] goes to each of the project
members, and is a great place to send questions and comments.
has been subjected to is limited, mostly to programs we have written. If you
encounter any issues or unexpected behaviour, the Github issue tracker is a good
way to raise them with the project members.

There are several specific issues we know about now, and want to point
out to avoid headaches for everyone involved:

1. In distributed mode, we strongly recommend that each process call
OnNext the same numbers of times before calling OnCompleted.
`OnNext` the same numbers of times before calling `OnCompleted`.
Several aspects of the graph manager logic consult the epoch numbers
associated with local inputs. Ideally, all process call OnNext the
same number of times; passing no parameters to OnNext causes the
associated with local inputs. Ideally, all process call `OnNext` the
same number of times; passing no parameters to `OnNext` causes the
epoch to advance with no input supplied, and is a great way to do this.

2. Naiad currently requires custom datatypes to be structs, rather
Expand All @@ -98,29 +116,31 @@ out to avoid headaches for everyone involved:
auto-generated Serialization/Deserialization code requires all of
your fields to be public and mutable (ie: not readonly). This is
because the generated code attempts to set these values on a newly
minted struct, and can�t do this without these settings. More
minted struct, and can't do this without these settings. More
details on writing custom serialization code in upcoming blog posts.

As Naiad is a research prototype, there are many parts of it that are
likely to change. We�ve tried hard to put interfaces in place that will
likely to change. We've tried hard to put interfaces in place that will
remain relatively stable, but there are certainly aspects of the
interfaces we expect will change. Information about programming examples,
the current state of the code, known problems, and upcoming fixes and
features should be available at the project web page:
interfaces we expect will change.

http://research.microsoft.com/Naiad/,
Some information about programming examples, the state of the code, known
problems, and fixes and features is available at the project web page:

or at the related blog:
http://research.microsoft.com/Naiad/,

http://bigdataatsvc.wordpress.com/
or at the related blog:

http://bigdataatsvc.wordpress.com/

Contacting us
-------------
Note, however, that the original authors no longer work at Microsoft Research,
and thus the website and the blog are not currently kept up to date.

If you have questions or comments, please feel welcome to email us at
# Contacting us

[email protected]
The former email address for questions (`[email protected]`) is no
longer actively monitored, as the original authors no longer work at Microsoft
Research. Raising issues on Github is the best way of getting in touch.

We'd love to hear your thoughts on the project. Thanks!

Expand Down