Code

Docs: How to Write a Data Pipeline?#247

Code

Merged

iesahin merged 159 commits intomainiesahin/xvc:mainfrom how-to-create-a-data-pipelineiesahin/xvc:how-to-create-a-data-pipelineCopy head branch name to clipboardDec 4, 2023

+1,309-153

Commits

Commits on Nov 16, 2023

• Add How to Create a Data Pipeline document

  iesahin

  committed
  64e19ad

  Copy full SHA for 64e19ad

• debug

  iesahin

  committed
  418d770

  Copy full SHA for 418d770

Commits on Nov 17, 2023

• fix hierarchical directories error

  iesahin

  committed
  0b42110

  Copy full SHA for 0b42110

• updated command

  iesahin

  committed
  9f8612e

  Copy full SHA for 9f8612e

• update permissions handling

  iesahin

  committed
  17f77f5

  Copy full SHA for 17f77f5

• debug carry_in

  iesahin

  committed
  ce7dc02

  Copy full SHA for ce7dc02

• debug

  iesahin

  committed
  f008a2d

  Copy full SHA for f008a2d

• debug

  iesahin

  committed
  5ff5298

  Copy full SHA for 5ff5298

• debug

  iesahin

  committed
  eb0baa4

  Copy full SHA for eb0baa4

• debug

  iesahin

  committed
  38d9978

  Copy full SHA for 38d9978

• debug

  iesahin

  committed
  f5a72a9

  Copy full SHA for f5a72a9

• debug

  iesahin

  committed
  1c790c3

  Copy full SHA for 1c790c3

• debug

  iesahin

  committed
  f73d283

  Copy full SHA for f73d283

• update

  iesahin

  committed
  af8cf81

  Copy full SHA for af8cf81

• debugged

  iesahin

  committed
  8f48316

  Copy full SHA for 8f48316

• Adding Chinese MNIST

  iesahin

  committed
  25482a0

  Copy full SHA for 25482a0

• Update commands

  iesahin

  committed
  98ad203

  Copy full SHA for 98ad203

• fix `

  iesahin

  committed
  a34ccd4

  Copy full SHA for a34ccd4

• as

  iesahin

  committed
  8fe3920

  Copy full SHA for 8fe3920

• added unzip commands

  iesahin

  committed
  57be37c

  Copy full SHA for 57be37c

• add ls

  iesahin

  committed
  a56b3fe

  Copy full SHA for a56b3fe

• add unzip to commands

  iesahin

  committed
  ec13327

  Copy full SHA for ec13327

• unzip q

  iesahin

  committed
  f322c07

  Copy full SHA for f322c07

• update

  iesahin

  committed
  e05c7b5

  Copy full SHA for e05c7b5

• update xfl

  iesahin

  committed
  808ac7f

  Copy full SHA for 808ac7f

• update

  iesahin

  committed
  817573a

  Copy full SHA for 817573a

• update

  iesahin

  committed
  fb20e4f

  Copy full SHA for fb20e4f

• update

  iesahin

  committed
  acc6746

  Copy full SHA for acc6746

• added copy

  iesahin

  committed
  a131f67

  Copy full SHA for a131f67

• increase command timeout

  iesahin

  committed
  c1d398f

  Copy full SHA for c1d398f

• update

  iesahin

  committed
  b500fb5

  Copy full SHA for b500fb5

• update

  iesahin

  committed
  ff08594

  Copy full SHA for ff08594

• update

  iesahin

  committed
  a9855c5

  Copy full SHA for a9855c5

• update

  iesahin

  committed
  4f80e44

  Copy full SHA for 4f80e44

• update

  iesahin

  committed
  9c6b72d

  Copy full SHA for 9c6b72d

• update

  iesahin

  committed
  3b59b61

  Copy full SHA for 3b59b61

• update

  iesahin

  committed
  b7c4539

  Copy full SHA for b7c4539

• update

  iesahin

  committed
  dc19d09

  Copy full SHA for dc19d09

• shell script

  iesahin

  committed
  b9bd28c

  Copy full SHA for b9bd28c

• add script

  iesahin

  committed
  7c2b0b2

  Copy full SHA for 7c2b0b2

• update

  iesahin

  committed
  942e7b9

  Copy full SHA for 942e7b9

• update

  iesahin

  committed
  3697e9e

  Copy full SHA for 3697e9e

• debug

  iesahin

  committed
  d1373f3

  Copy full SHA for d1373f3

• debug flag

  iesahin

  committed
  a1a2c1b

  Copy full SHA for a1a2c1b

• update

  iesahin

  committed
  ac5333b

  Copy full SHA for ac5333b

• update

  iesahin

  committed
  85cb109

  Copy full SHA for 85cb109

• added name only option to copy

  iesahin

  committed
  560dbe8

  Copy full SHA for 560dbe8

• fix compilation

  iesahin

  committed
  f2e854b

  Copy full SHA for f2e854b

• update script

  iesahin

  committed
  983459f

  Copy full SHA for 983459f

• update

  iesahin

  committed
  d646abb

  Copy full SHA for d646abb

• fix flag

  iesahin

  committed
  cdd0952

  Copy full SHA for cdd0952

• update copy debug

  iesahin

  committed
  7d28e60

  Copy full SHA for 7d28e60

• update script

  iesahin

  committed
  5eca5c7

  Copy full SHA for 5eca5c7

• update script

  iesahin

  committed
  b753db0

  Copy full SHA for b753db0

• update

  iesahin

  committed
  25d2582

  Copy full SHA for 25d2582

• update

  iesahin

  committed
  ce77ea7

  Copy full SHA for ce77ea7

• update

  iesahin

  committed
  9ef79b7

  Copy full SHA for 9ef79b7

• upd

  iesahin

  committed
  e3ac24c

  Copy full SHA for e3ac24c

• loop

  iesahin

  committed
  c43a538

  Copy full SHA for c43a538

• up

  iesahin

  committed
  1dee99d

  Copy full SHA for 1dee99d

• update cmd

  iesahin

  committed
  0fc67ea

  Copy full SHA for 0fc67ea

• update

  iesahin

  committed
  b8e8ed3

  Copy full SHA for b8e8ed3

• update

  iesahin

  committed
  437863c

  Copy full SHA for 437863c

• update

  iesahin

  committed
  1900ff6

  Copy full SHA for 1900ff6

Commits on Nov 18, 2023

• Update commands

  iesahin

  committed
  0e6e1df

  Copy full SHA for 0e6e1df

• update

  iesahin

  committed
  069ca87

  Copy full SHA for 069ca87

• update

  iesahin

  committed
  e610c8c

  Copy full SHA for e610c8c

• update command

  iesahin

  committed
  48cc391

  Copy full SHA for 48cc391

Commits on Nov 19, 2023

• added pipeline steps

  iesahin

  committed
  812c873

  Copy full SHA for 812c873

• update

  iesahin

  committed
  bac9ea7

  Copy full SHA for bac9ea7

• update

  iesahin

  committed
  21ba65b

  Copy full SHA for 21ba65b

• try once more

  iesahin

  committed
  675b9e8

  Copy full SHA for 675b9e8

• update

  iesahin

  committed
  b167bd7

  Copy full SHA for b167bd7

Commits on Nov 21, 2023

• Updated to handle symlinks

  iesahin

  committed
  614d049

  Copy full SHA for 614d049

Commits on Nov 22, 2023

• updating recheck

  iesahin

  committed
  1d6b567

  Copy full SHA for 1d6b567

• better error handling in walker

  iesahin

  committed
  03c5393

  Copy full SHA for 03c5393

• update

  iesahin

  committed
  0a50fb5

  Copy full SHA for 0a50fb5

• Update data pipeline

  iesahin

  committed
  27c8911

  Copy full SHA for 27c8911

• update recheck cmd

  iesahin

  committed
  c7f813f

  Copy full SHA for c7f813f

• watch

  iesahin

  committed
  5cbb7c6

  Copy full SHA for 5cbb7c6

• Update recheck

  iesahin

  committed
  7ca837a

  Copy full SHA for 7ca837a

• recheck

  iesahin

  committed
  2a12478

  Copy full SHA for 2a12478

• update commands

  iesahin

  committed
  bb9ef49

  Copy full SHA for bb9ef49

• debug recheck

  iesahin

  committed
  9c6a0bc

  Copy full SHA for 9c6a0bc

• Fixed recheck method bug

  iesahin

  committed
  d2439d0

  Copy full SHA for d2439d0

• rename

  iesahin

  committed
  3863bd9

  Copy full SHA for 3863bd9

• update

  iesahin

  committed
  2190a8f

  Copy full SHA for 2190a8f

• cmd

  iesahin

  committed
  73cc285

  Copy full SHA for 73cc285

• update

  iesahin

  committed
  9a71059

  Copy full SHA for 9a71059

• track all

  iesahin

  committed
  e94ee65

  Copy full SHA for e94ee65

• update prefix

  iesahin

  committed
  731487f

  Copy full SHA for 731487f

• remove inner

  iesahin

  committed
  c97d1c7

  Copy full SHA for c97d1c7

• update

  iesahin

  committed
  e4ebce4

  Copy full SHA for e4ebce4

• debug

  iesahin

  committed
  e77b852

  Copy full SHA for e77b852

• index

  iesahin

  committed
  3305488

  Copy full SHA for 3305488

Commits on Nov 23, 2023

• Added xvc_walker::PathSync to sync operations on paths

  iesahin

  committed
  3925ef3

  Copy full SHA for 3925ef3

• Test sync

  iesahin

  committed
  fbc6a68

  Copy full SHA for fbc6a68

• Update cmd

  iesahin

  committed
  963fc4b

  Copy full SHA for 963fc4b

• update

  iesahin

  committed
  ee7e220

  Copy full SHA for ee7e220

• update cmd

  iesahin

  committed
  84725c3

  Copy full SHA for 84725c3

• remove watch in loop

  iesahin

  committed
  21f27e2

  Copy full SHA for 21f27e2

• update

  iesahin

  committed
  7a5b910

  Copy full SHA for 7a5b910

• update walker notify

  iesahin

  committed
  4280f21

  Copy full SHA for 4280f21

• update

  iesahin

  committed
  50a4823

  Copy full SHA for 50a4823

• update

  iesahin

  committed
  8a6b6b3

  Copy full SHA for 8a6b6b3

• explanation

  iesahin

  committed
  20ccbb7

  Copy full SHA for 20ccbb7

Commits on Nov 26, 2023

• Added another sections

  iesahin

  committed
  e697fa1

  Copy full SHA for e697fa1

• fix command errors

  iesahin

  committed
  543b6c1

  Copy full SHA for 543b6c1

• update

  iesahin

  committed
  4102e71

  Copy full SHA for 4102e71

• added requirements file

  iesahin

  committed
  4c7cd15

  Copy full SHA for 4c7cd15

• debug

  iesahin

  committed
  0097b3b

  Copy full SHA for 0097b3b

• update

  iesahin

  committed
  950f4ec

  Copy full SHA for 950f4ec

Commits on Nov 28, 2023

• fix pipeline

  iesahin

  committed
  132f34a

  Copy full SHA for 132f34a

• update

  iesahin

  committed
  cdb56ad

  Copy full SHA for cdb56ad

• update

  iesahin

  committed
  2d3a42d

  Copy full SHA for 2d3a42d

• save images and classes

  iesahin

  committed
  9285f2a

  Copy full SHA for 9285f2a

• updated requirements

  iesahin

  committed
  b94144a

  Copy full SHA for b94144a

• update

  iesahin

  committed
  f6836ac

  Copy full SHA for f6836ac

• update python cmds

  iesahin

  committed
  497ad1e

  Copy full SHA for 497ad1e

• update installation command

  iesahin

  committed
  eb374b5

  Copy full SHA for eb374b5

• update

  iesahin

  committed
  80a317a

  Copy full SHA for 80a317a

Commits on Nov 29, 2023

• update

  iesahin

  committed
  6b70587

  Copy full SHA for 6b70587

• update

  iesahin

  committed
  cf41468

  Copy full SHA for cf41468

• update

  iesahin

  committed
  d899f99

  Copy full SHA for d899f99

• update

  iesahin

  committed
  35490ea

  Copy full SHA for 35490ea

• update

  iesahin

  committed
  18edee4

  Copy full SHA for 18edee4

• Update

  iesahin

  committed
  04fb100

  Copy full SHA for 04fb100

Commits on Nov 30, 2023

• Add training script

  iesahin

  committed
  bb7c266

  Copy full SHA for bb7c266

• fix pipeline commands

  iesahin

  committed
  11f1b2e

  Copy full SHA for 11f1b2e

Commits on Dec 1, 2023

• update output commands

  iesahin

  committed
  4669ae5

  Copy full SHA for 4669ae5

• pipeline again

  iesahin

  committed
  16e2d70

  Copy full SHA for 16e2d70

• fix deps and identities

  iesahin

  committed
  8a5919a

  Copy full SHA for 8a5919a

• update

  iesahin

  committed
  d3758ce

  Copy full SHA for d3758ce

• update

  iesahin

  committed
  dd0de8b

  Copy full SHA for dd0de8b

• update

  iesahin

  committed
  1c0d285

  Copy full SHA for 1c0d285

• update

  iesahin

  committed
  606531a

  Copy full SHA for 606531a

• update

  iesahin

  committed
  cd4c189

  Copy full SHA for cd4c189

• run the pipeline

  iesahin

  committed
  ff471ce

  Copy full SHA for ff471ce

• requirements

  iesahin

  committed
  24572ea

  Copy full SHA for 24572ea

• update requirements

  iesahin

  committed
  c0544e3

  Copy full SHA for c0544e3

• update requirements

  iesahin

  committed
  f3725a3

  Copy full SHA for f3725a3

• update train command

  iesahin

  committed
  ef53b7d

  Copy full SHA for ef53b7d

• update

  iesahin

  committed
  ec6d350

  Copy full SHA for ec6d350

• update

  iesahin

  committed
  f7c1e37

  Copy full SHA for f7c1e37

• update

  iesahin

  committed
  35b68c8

  Copy full SHA for 35b68c8

• update

  iesahin

  committed
  4cffa82

  Copy full SHA for 4cffa82

• Fix train.py error

  iesahin

  committed
  c36ef3c

  Copy full SHA for c36ef3c

• update the train script

  iesahin

  committed
  308124b

  Copy full SHA for 308124b

• Updated train script

  iesahin

  committed
  6e4547a

  Copy full SHA for 6e4547a

Commits on Dec 2, 2023

• Update docs

  iesahin

  committed
  25d06ca

  Copy full SHA for 25d06ca

• remove how to from the runs

  iesahin

  committed
  2564216

  Copy full SHA for 2564216

• Latest version

  iesahin

  committed
  5702d9c

  Copy full SHA for 5702d9c

• Update after tests

  iesahin

  committed
  5646963

  Copy full SHA for 5646963

• Merge remote-tracking branch 'origin/main' into how-to-create-a-data-pipeline

  iesahin

  committed
  2acf682

  Copy full SHA for 2acf682

• update

  iesahin

  committed
  d0f9e0b

  Copy full SHA for d0f9e0b

Commits on Dec 3, 2023

• update

  iesahin

  committed
  870149c

  Copy full SHA for 870149c

• add images

  iesahin

  committed
  31656b2

  Copy full SHA for 31656b2

Commits on Dec 4, 2023

• Update dag for sorting the stmts

  iesahin

  committed
  46c7a4c

  Copy full SHA for 46c7a4c

• added output to pipeline

  iesahin

  committed
  3ce832a

  Copy full SHA for 3ce832a