sotastream package

Subpackages

• sotastream.augmentors package
  • Submodules
    • sotastream.augmentors.augmentors module
      • Append()
      • Copy()
      • CopySource()
      • DataSource()
      • Identity()
      • JustSourceTarget()
      • Mixer
      • Multiply()
      • SPMDecoder()
      • SPMEncoder()
      • Tagger()
      • ToLower()
      • ToTitle()
      • ToUpper()
      • UTF8File()
      • canBeLowercased()
      • canBeUppercased()
      • enumerate_files()
  • Module contents
• sotastream.filters package
  • Submodules
    • sotastream.filters.filters module
      • BitextFilter()
      • MatchFilter()
      • RegexFilter()
      • SkipBlanks()
  • Module contents
• sotastream.pipelines package
  • Submodules
    • sotastream.pipelines.base module
      • DocumentPipeline
      • Pipeline
    • sotastream.pipelines.default module
      • DefaultPipeline
    • sotastream.pipelines.example_pipeline module
      • ExamplePipeline
      • LowerCase()
      • ReadAndAugment()
      • TagData()
      • TitleCase()
    • sotastream.pipelines.mtdata_pipeline module
      • MTDataPipeline
      • MTDataSource()
    • sotastream.pipelines.multistream_pipeline module
      • MultiStreamPipeline
  • Module contents
• sotastream.utils package
  • Submodules
    • sotastream.utils.phrases module
      • PhraseSpanExtractor
    • sotastream.utils.split module
      • compute_md5()
      • smart_open()
      • split_file_into_chunks()
      • split_native()
      • split_subshell()
  • Module contents

Submodules

• sotastream.cli module
  • add_global_args()
  • adjustSeed()
  • main()
  • maybe_split_files()
  • run_pipeline_process()
• sotastream.data module
  • Line
    • Line.append()
    • Line.fields
    • Line.join()

Module contents

class sotastream.Defaults

Bases: object

Default values for pipeline arguments

BUFFER_SIZE = 1000000

DOC_PROB = 0.0

DOC_PROB_PARALLEL = 0.0

DOC_SEPARATOR = ' <eos>'

MAX_TOKENS = 250

NUM_PROCESSES = 16

QUEUE_BUFFER_SIZE = 10000

QUIET = False

SAMPLE_FILE = None

SAMPLE_LENGTH = True

SEED = 0

SEPARATOR = ' '

SHUFFLE = True

SPM_MODEL = None